# rudrabha/wav2lip architecture

> Architecture diagram of the GitHub repository rudrabha/wav2lip, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rudrabha/wav2lip
- Repository: https://github.com/rudrabha/wav2lip

## Overview

##### Main Components
a. Data Ingestion & Preprocessing
- Raw video + audio files
- `preprocess.py` reads `filelists/` → generates face crops (`.jpg`) + audio (`.wav`) under `lrs2_preprocessed/`
b. Face Detection Module
- `face_detection/api.py` & `face_detection/detection/sfd` (S3FD)
c. Expert Discriminator (SyncNet) Training
- `color_syncnet_train.py`
- Uses PyTorch model in `models/syncnet.py`
- Trains SyncNet (“expert lip‐sync discriminator”) on preprocessed data → saves checkpoints in `checkpoints/`
d. Wav2Lip Model Training
- Two scripts:
– `wav2lip_train.py` (base model)
– `hq_wav2lip_train.py` (with visual‐quality GAN discriminator)
- Underlying model in `models/wav2lip.py`
- Reads SyncNet checkpoint, preprocessed data → outputs Wav2Lip checkpoints
e. Inference Pipeline
- `inference.py`
- Inputs: arbitrary video (`--face`), audio (`--audio`), Wav2Lip checkpoint
- Invokes face detection → crops + audio → Wav2Lip model → merges generated frames + audio → writes to `results/`
f. Evaluation & Metrics
- `evaluation/` folder
- Scripts to compute LSE‐D, LSE‐C, PSNR, SSIM, etc.
- Reads generated videos from `results/` and reference lists from `evaluation/test_filelists/`
g. Checkpoints & Results Storage
- `checkpoints/` stores all training checkpoints
- `results/` stores inference outputs

##### Architectural Patterns & Design Principles
- Modular separation:
– Data preparation (`preprocess.py`)
– Model training (SyncNet vs Wav2Lip)
– Inference (`inference.py`)
– Evaluation (`evaluation/`)
- Reusable components: face detection API isolated under `face_detection/`
- Centralized hyperparameters: `hparams.py`
- Extensibility points: new discriminators, custom datasets via `filelists/`

##### Technologies
- Python 3.6, PyTorch, ffmpeg
- Face detection: S3FD (from `face_detection/detection/sfd`)
- Models:
– `models/syncnet.py`
– `models/wav2lip.py`
– (Optional) Conv blocks in `models/conv.py`
- External storage: Google Drive links for pretrained weights

##### Don’t Overthink—Separate Into As Many Logical Components As Possible
- Treat each script or directory as its own module if it encapsulates a distinct responsibility
- Explicitly show:
– Data inputs/outputs (raw folders, preprocessed folders)
– Model artifacts (checkpoint files)
– Evaluation metrics outputs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        RawData["Raw Video/Audio"]:::data
        Filelists["filelists/"]:::data
        Preprocess["preprocess.py"]:::compute
        Preprocessed["lrs2_preprocessed/"]:::data

        RawData -->|input| Preprocess
        Filelists -->|datasets| Preprocess
        Preprocess -->|outputs| Preprocessed
    end

    %% Face Detection Module
    subgraph "Face Detection Module"
        FaceDetectAPI["face_detection/api.py"]:::compute
        SFD["face_detection/detection/sfd/"]:::external

        FaceDetectAPI -->|uses| SFD
    end

    %% Model Training
    subgraph "Model Training"
        SyncTrain["color_syncnet_train.py"]:::compute
        SyncNetModel["models/syncnet.py"]:::compute
        SyncCheckpoint["checkpoints/"]:::data

        Wav2LipTrain["wav2lip_train.py"]:::compute
        HQTrain["hq_wav2lip_train.py"]:::compute
        Wav2LipModel["models/wav2lip.py"]:::compute
        Wav2LipCheckpoint["checkpoints/"]:::data

        Hparams["hparams.py"]:::compute

        Preprocessed -->|train data| SyncTrain
        SyncNetModel -->|model| SyncTrain
        Hparams -->|hyperparams| SyncTrain
        SyncTrain -->|checkpoint| SyncCheckpoint

        Preprocessed -->|train data| Wav2LipTrain
        SyncCheckpoint -->|syncnet ckpt| Wav2LipTrain
        Hparams -->|hyperparams| Wav2LipTrain
        Wav2LipModel -->|model| Wav2LipTrain
        Wav2LipTrain -->|checkpoint| Wav2LipCheckpoint

        Preprocessed -->|train data| HQTrain
        SyncCheckpoint -->|syncnet ckpt| HQTrain
        Hparams -->|hyperparams| HQTrain
        Wav2LipModel -->|model| HQTrain
        HQTrain -->|checkpoint| Wav2LipCheckpoint
    end

    %% Inference Pipeline
    subgraph "Inference Pipeline"
        Inference["inference.py"]:::compute
        RawDataInput["Raw Video/Audio"]:::data
        Inference -->|uses| FaceDetectAPI
        Inference -->|uses| Wav2LipModel
        RawDataInput -->|input| Inference
        Wav2LipCheckpoint -->|ckpt| Inference
        Inference -->|output| Results["results/"]:::data
    end

    %% Evaluation & Metrics
    subgraph "Evaluation & Metrics"
        ResultsNode["results/"]:::data
        GenVideos["evaluation/gen_videos_from_filelist.py"]:::compute
        RealVideos["evaluation/real_videos_inference.py"]:::compute
        Scores["evaluation/scores_LSE/"]:::data
        TestFilelists["evaluation/test_filelists/"]:::data

        ResultsNode -->|generated| GenVideos
        ResultsNode -->|generated| RealVideos
        TestFilelists -->|references| GenVideos
        TestFilelists -->|references| RealVideos
        GenVideos -->|metrics| Scores
        RealVideos -->|metrics| Scores
    end

    %% Click Events
    click Preprocess "https://github.com/rudrabha/wav2lip/blob/master/preprocess.py"
    click Filelists "https://github.com/rudrabha/wav2lip/tree/master/filelists/"
    click FaceDetectAPI "https://github.com/rudrabha/wav2lip/blob/master/face_detection/api.py"
    click SFD "https://github.com/rudrabha/wav2lip/tree/master/face_detection/detection/sfd/"
    click SyncTrain "https://github.com/rudrabha/wav2lip/blob/master/color_syncnet_train.py"
    click SyncNetModel "https://github.com/rudrabha/wav2lip/blob/master/models/syncnet.py"
    click SyncCheckpoint "https://github.com/rudrabha/wav2lip/tree/master/checkpoints/"
    click Wav2LipTrain "https://github.com/rudrabha/wav2lip/blob/master/wav2lip_train.py"
    click HQTrain "https://github.com/rudrabha/wav2lip/blob/master/hq_wav2lip_train.py"
    click Wav2LipModel "https://github.com/rudrabha/wav2lip/blob/master/models/wav2lip.py"
    click Wav2LipCheckpoint "https://github.com/rudrabha/wav2lip/tree/master/checkpoints/"
    click Inference "https://github.com/rudrabha/wav2lip/blob/master/inference.py"
    click Results "https://github.com/rudrabha/wav2lip/tree/master/results/"
    click EvalFolder "https://github.com/rudrabha/wav2lip/tree/master/evaluation/"
    click GenVideos "https://github.com/rudrabha/wav2lip/blob/master/evaluation/gen_videos_from_filelist.py"
    click RealVideos "https://github.com/rudrabha/wav2lip/blob/master/evaluation/real_videos_inference.py"
    click Scores "https://github.com/rudrabha/wav2lip/tree/master/evaluation/scores_LSE/"
    click TestFilelists "https://github.com/rudrabha/wav2lip/tree/master/evaluation/test_filelists/"
    click Hparams "https://github.com/rudrabha/wav2lip/blob/master/hparams.py"

    %% Styles
    classDef data fill:#f9f,stroke:#333,stroke-width:1px
    classDef compute fill:#bbf,stroke:#333,stroke-width:1px
    classDef external fill:#fea,stroke:#333,stroke-width:1px
```
