How rudrabha/wav2lip works
Main Components
a. Data Ingestion & Preprocessing
• Raw video + audio files
• preprocess.py reads filelists/ → generates face crops (.jpg) + audio (.wav) under lrs2_preprocessed/
b. Face Detection Module
• face_detection/api.py & face_detection/detection/sfd (S3FD)
c. Expert Discriminator (SyncNet) Training
• color_syncnet_train.py
• Uses PyTorch model in models/syncnet.py
• Trains SyncNet (“expert lip‐sync discriminator”) on preprocessed data → saves checkpoints in checkpoints/
d. Wav2Lip Model Training
• Two scripts:
– wav2lip_train.py (base model)
– hq_wav2lip_train.py (with visual‐quality GAN discriminator)
• Underlying model in models/wav2lip.py
• Reads SyncNet checkpoint, preprocessed data → outputs Wav2Lip checkpoints
e. Inference Pipeline
• inference.py
• Inputs: arbitrary video (--face), audio (--audio), Wav2Lip checkpoint
• Invokes face detection → crops + audio → Wav2Lip model → merges generated frames + audio → writes to results/
f. Evaluation & Metrics
• evaluation/ folder
• Scripts to compute LSE‐D, LSE‐C, PSNR, SSIM, etc.
• Reads generated videos from results/ and reference lists from evaluation/test_filelists/
g. Checkpoints & Results Storage
• checkpoints/ stores all training checkpoints
• results/ stores inference outputs
Architectural Patterns & Design Principles
• Modular separation:
– Data preparation (preprocess.py)
– Model training (SyncNet vs Wav2Lip)
– Inference (inference.py)
– Evaluation (evaluation/)
• Reusable components: face detection API isolated under face_detection/
• Centralized hyperparameters: hparams.py
• Extensibility points: new discriminators, custom datasets via filelists/
Technologies
• Python 3.6, PyTorch, ffmpeg
• Face detection: S3FD (from face_detection/detection/sfd)
• Models:
– models/syncnet.py
– models/wav2lip.py
– (Optional) Conv blocks in models/conv.py
• External storage: Google Drive links for pretrained weights
Don’t Overthink—Separate Into As Many Logical Components As Possible
• Treat each script or directory as its own module if it encapsulates a distinct responsibility
• Explicitly show:
– Data inputs/outputs (raw folders, preprocessed folders)
– Model artifacts (checkpoint files)
– Evaluation metrics outputs