# antonibigata/keysync architecture

> Architecture diagram of the GitHub repository antonibigata/keysync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antonibigata/keysync
- Repository: https://github.com/antonibigata/keysync

## Overview

##### Project Type and Purpose
- KeySync is a research‐grade, full-stack lip-synchronization pipeline comprising:
- Data preparation & preprocessing (video + audio)
- Two-stage deep learning models (Keyframe & Interpolation)
- Inference scripts, training loops, and evaluation modules
- Its purpose is to take raw video and audio, process them through embedding and masking pipelines, run two sequential neural models, and output leakage-free, high-resolution lip-synced videos.

##### High-Level File Structure
- Top-Level:
- configs/         ← YAML configurations for training/inference
- sgm/             ← Core model code (autoencoder & diffusion)
- evaluation/      ← LipScore metric & audio–visual speech recognition (AVSR)
- scripts/         ← Shell and Python scripts to orchestrate preprocessing, inference, training
- main.py & train_*.sh ← Entrypoints for inference & training
- Models & Modules under sgm/:
- sgm/models/     ← Neural network definitions (autoencoder, diffusion)
- sgm/modules/    ← Building blocks (attention, temporal AE, loss, sampling)
- sgm/data/       ← PyTorch DataModules and datasets
- sgm/inference/  ← Inference API & helpers
- evaluation/auto_avsr/: separate AVSR benchmark to evaluate lip-sync quality
- notebooks/: exploratory analysis

##### Key Components
a. Data Ingestion & Preprocessing
- Video inputs → Face landmark detection (Mediapipe/Retinaface) → Video embeddings via autoencoder
- Audio inputs → Feature extraction (WavLM, HuBERT) → Audio embeddings
b. Two-Stage Models
1. Keyframe Model (sgm/models/autoencoder & sgm/modules/autoencoding)
- Learns to predict masked lip keyframes
2. Interpolation Model (sgm/models/diffusion & sgm/modules/diffusionmodules)
- Denoising diffusion to interpolate smooth lip motion between keyframes
c. Orchestration Layers
- PyTorch Lightning trainers (sgm/callbacks & lightning.py)
- Configuration management via OmegaConf (configs/*.yaml)
- Shell scripts for end-to-end pipelines (scripts/infer_raw.sh, train_keyframe.sh, train_interpolation.sh)
d. Evaluation & Metrics
- LipScore module (evaluation/lipscore.py)
- Optional AVSR evaluation (evaluation/auto_avsr/)

##### Relationships and Data Flow
- Video embeddings and audio embeddings converge into Keyframe stage → output keyframe predictions
- Keyframe outputs feed into Interpolation stage → final video reconstruction
- The reconstruction method invokes the trained autoencoder decoder to produce pixel frames
- Evaluation components take generated video + ground truth video as inputs → LipScore and AVSR metrics

##### Architectural Patterns & Design Principles
- Modular ML pipeline: clear separation of data, model, and inference code
- Two-stage “coarse-to-fine” framework (keyframe + diffusion)
- Configurable via Hydra/OmegaConf for reproducibility
- PyTorch Lightning for standardized training loops and callbacks
- Extensibility: separate sgm/modules for plugging new loss functions or diffusion strategies

##### Essential Elements Checklist
[ ] Clear labels for each component
[ ] References to config files triggering each stage

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Preprocessing Layer"
        A1["Raw Video"]:::data
        A2["Raw Audio"]:::data
        LDet["Landmarks Extraction"]:::data
        click LDet "https://github.com/antonibigata/keysync/blob/main/scripts/util/landmarks_extractor.py"
        GLand["Generate Landmarks"]:::data
        click GLand "https://github.com/antonibigata/keysync/blob/main/scripts/util/gen_landmarks.py"
        AEmbGet["Get Audio Embeddings"]:::data
        click AEmbGet "https://github.com/antonibigata/keysync/blob/main/scripts/util/get_audio_embeddings.py"
    end

    subgraph "Embedding Extraction Layer"
        VE["Video to Latent (VAE)"]:::model
        click VE "https://github.com/antonibigata/keysync/blob/main/scripts/util/video_to_latent.py"
        AEQ["Autoencoder Model"]:::model
        click AEQ "https://github.com/antonibigata/keysync/blob/main/sgm/models/autoencoder.py"
        TAe["Temporal AE Module"]:::model
        click TAe "https://github.com/antonibigata/keysync/blob/main/sgm/modules/autoencoding/temporal_ae.py"
        WavLMmod["WavLM Audio Module"]:::model
        click WavLMmod "https://github.com/antonibigata/keysync/blob/main/scripts/util/audio/WavLM_modules.py"
    end

    subgraph "Model Inference Layer"
        subgraph "Stage 1: Keyframe Model"
            S1["Keyframe Predictor"]:::model
            click S1 "https://github.com/antonibigata/keysync/blob/main/sgm/models/autoencoder.py"
        end
        subgraph "Stage 2: Interpolation Model"
            S2["Diffusion Interpolator"]:::model
            click S2 "https://github.com/antonibigata/keysync/blob/main/sgm/models/diffusion.py"
        end
    end

    subgraph "Post-processing & Reconstruction Layer"
        Decode["VAE Decoder to Frames"]:::model
        click Decode "https://github.com/antonibigata/keysync/blob/main/scripts/util/vae_wrapper.py"
        VidOut["Final Video"]:::data
    end

    subgraph "Evaluation Layer"
        LipScore["LipScore Metric"]:::eval
        click LipScore "https://github.com/antonibigata/keysync/blob/main/evaluation/lipscore.py"
        AVSR["AVSR Evaluation"]:::eval
        click AVSR "https://github.com/antonibigata/keysync/blob/main/evaluation/auto_avsr/eval.py"
    end

    subgraph "Control Plane"
        Configs["YAML Configs"]:::orchestration
        click Configs "https://github.com/antonibigata/keysync/blob/main/configs/example_training/interpolation/interpolation_dub.yaml"
        Scripts["Shell & Python Scripts"]:::orchestration
        click Scripts "https://github.com/antonibigata/keysync/blob/main/scripts/infer_raw.sh"
    end

    %% Data Flow
    A1 --> LDet --> GLand --> VE
    A2 --> AEmbGet --> WavLMmod
    VE --> S1
    WavLMmod --> S1
    S1 --> KFOut["Keyframe Outputs"]:::model
    KFOut --> S2
    S2 --> FinalLatent["Interpolated Latents"]:::model
    FinalLatent --> Decode --> VidOut

    %% Evaluation Flow
    VidOut --> LipScore
    VidOut --> AVSR

    %% Control Plane Flow
    Configs --> S1
    Configs --> S2
    Scripts --> LDet
    Scripts --> S1
    Scripts --> S2
    Scripts --> VidOut

    classDef data fill:#D0E7FF,stroke:#0B5394,color:#0B5394;
    classDef model fill:#D5E8D4,stroke:#38761D,color:#38761D;
    classDef orchestration fill:#FFE6CC,stroke:#E69138,color:#E69138;
    classDef eval fill:#E1D5E7,stroke:#674EA7,color:#674EA7;
```
