# antonibigata/keyface_cvpr architecture

> Architecture diagram of the GitHub repository antonibigata/keyface_cvpr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/antonibigata/keyface_cvpr
- Repository: https://github.com/antonibigata/keyface_cvpr

## Overview

Based on our analysis of both the file tree and the README, KeyFace is a two-stage diffusion-based framework for audio-driven facial animation over long sequences. Its purpose is to efficiently generate keyframes based on audio and identity input and then use an interpolation model to fill in the gaps and ensure temporal coherence in the final animation. Given its scope, the project is a full-stack “research-to-production” pipeline where data ingestion, pre-processing, model training/inference, and evaluation are clearly separated into distinct modules.

##### Main Components
- Data Input and Preprocessing:
- A “Preprocessing / Embedding Extraction” module where raw video and audio data are converted into embeddings.

- Two-Stage Modeling Pipeline:
- Stage 1 – Keyframe Model:
‣ Represent this as a dedicated component (with configuration files under configs/example_training/keyframes and related training scripts such as train_keyframe.sh).
‣ Indicate that it takes as input audio embeddings (and an identity frame) and outputs a sparse set of keyframes.
- Stage 2 – Interpolation Model:
‣ Represent this as another distinct component (with configurations under configs/example_training/interpolation and its own training script train_interpolation.sh).
‣ Show that it accepts keyframes as input and produces a dense frame sequence for smooth animation.

- Model Infrastructure and Training:
- Components such as “configs,” “scripts,” and “sgm modules,” which include diffusion modules, autoencoding, and loss functions (accessible under sgm/ and sgm/modules).

- Inference and Post-processing:
- A storage or “Output Folder” node where generated animations are saved.

- Evaluation:
- An “Evaluation Module” which covers both lip-sync quality (via the LipScore evaluation in evaluation/lipscore.py) and any additional metrics.
- Integrations with external libraries for face detection and alignment.

##### Relationships & Data Flow
- Raw data → Preprocessing (embedding extraction) → Keyframe Model (Stage 1) → Interpolation Model (Stage 2) → Final Animation Output.

##### Architectural Patterns and Key Technologies
- Use of a modular, two-stage pipeline as a key architectural pattern.
- Out the separation of concerns: data handling, model training, inference, and evaluation are clearly distinct.
- Technologies explicitly:
- Python 3.11, PyTorch (with CUDA), Conda for environment management,
- Diffusion and autoencoding modules under sgm/,
- External repositories for face detection/alignment in evaluation.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Input and Preprocessing
    subgraph "Data Input and Preprocessing"
        DS["Data Sources (.wav, .mp4)"]:::dataHandling
        FC["Filelist Creator"]:::dataHandling
        AE["Audio Embedding Extraction"]:::dataHandling
        VT["Video-to-Latent Conversion"]:::dataHandling
    end

    DS -->|"to"| FC
    FC -->|"generates"| AE
    FC -->|"generates"| VT

    %% Two-Stage Modeling Pipeline
    subgraph "Two-Stage Modeling Pipeline"
        KC["Keyframe Config"]:::modelTraining
        KT["Keyframe Training Script"]:::modelTraining
        KM["Keyframe Model"]:::modelTraining
        IC["Interpolation Config"]:::modelTraining
        IT["Interpolation Training Script"]:::modelTraining
        IM["Interpolation Model"]:::modelTraining
    end

    AE -->|"provides audio embeddings"| KM
    VT -->|"provides identity frame"| KM
    KC -->|"configures"| KM
    KT -->|"trains"| KM
    KM -->|"keyframes feed"| IM
    IC -->|"configures"| IM
    IT -->|"trains"| IM

    %% Model Infrastructure & Training
    subgraph "Model Infrastructure & Training"
        Conf["Configurations"]:::modelTraining
        Scr["Scripts"]:::modelTraining
        DiffMod["Diffusion Modules"]:::modelTraining
        AutoEnc["Autoencoding Modules"]:::modelTraining
    end

    Conf -->|"supports"| KM
    Conf -->|"supports"| IM
    Scr -->|"supports"| KM
    Scr -->|"supports"| IM
    DiffMod -->|"integrates with"| KM
    DiffMod -->|"integrates with"| IM
    AutoEnc -->|"integrates with"| KM
    AutoEnc -->|"integrates with"| IM

    %% Inference and Post-processing
    subgraph "Inference and Post-processing"
        IRD["Inference Raw Data"]:::modelTraining
        IP["Inference Pipeline"]:::modelTraining
        Out["Output Folder (Generated Animation)"]:::modelTraining
    end

    IM -->|"input to"| IRD
    IRD -->|"processes into"| IP
    IP -->|"results saved to"| Out

    %% Evaluation
    subgraph "Evaluation"
        LipScore["LipSync Evaluation"]:::evaluation
        AutoAvsr["AV Evaluation"]:::evaluation
        Eval["Evaluation Module"]:::evaluation
    end

    Out -->|"evaluated by"| Eval
    LipScore -->|"feeds into"| Eval
    AutoAvsr -->|"feeds into"| Eval
    Eval -- "feedback" --> Conf

    %% Click Events
    click DS "https://github.com/antonibigata/keyface_cvpr/tree/main/assets/media"
    click FC "https://github.com/antonibigata/keyface_cvpr/blob/main/scripts/util/create_filelist.py"
    click AE "https://github.com/antonibigata/keyface_cvpr/blob/main/scripts/util/get_audio_embeddings.py"
    click VT "https://github.com/antonibigata/keyface_cvpr/blob/main/scripts/util/video_to_latent.py"
    click KC "https://github.com/antonibigata/keyface_cvpr/blob/main/configs/example_training/keyframes/keyframes_base.yaml"
    click KT "https://github.com/antonibigata/keyface_cvpr/blob/main/train_keyframe.sh"
    click IC "https://github.com/antonibigata/keyface_cvpr/blob/main/configs/example_training/interpolation/interpolation_base.yaml"
    click IT "https://github.com/antonibigata/keyface_cvpr/blob/main/train_interpolation.sh"
    click Conf "https://github.com/antonibigata/keyface_cvpr/tree/main/configs/"
    click Scr "https://github.com/antonibigata/keyface_cvpr/tree/main/scripts/"
    click DiffMod "https://github.com/antonibigata/keyface_cvpr/tree/main/sgm/modules/diffusionmodules/"
    click AutoEnc "https://github.com/antonibigata/keyface_cvpr/tree/main/sgm/modules/autoencoding/"
    click IRD "https://github.com/antonibigata/keyface_cvpr/blob/main/scripts/infer_raw_data.sh"
    click IP "https://github.com/antonibigata/keyface_cvpr/blob/main/scripts/inference.sh"
    click LipScore "https://github.com/antonibigata/keyface_cvpr/blob/main/evaluation/lipscore.py"
    click AutoAvsr "https://github.com/antonibigata/keyface_cvpr/tree/main/evaluation/auto_avsr/"

    %% Styles
    classDef dataHandling fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef modelTraining fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#FFA500,stroke:#333,stroke-width:2px;
```
