How antonibigata/keysync works
Project Type and Purpose
• KeySync is a research‐grade, full-stack lip-synchronization pipeline comprising:
• Data preparation & preprocessing (video + audio)
• Two-stage deep learning models (Keyframe & Interpolation)
• Inference scripts, training loops, and evaluation modules
• Its purpose is to take raw video and audio, process them through embedding and masking pipelines, run two sequential neural models, and output leakage-free, high-resolution lip-synced videos.
High-Level File Structure
• Top-Level:
• configs/ ← YAML configurations for training/inference
• sgm/ ← Core model code (autoencoder & diffusion)
• evaluation/ ← LipScore metric & audio–visual speech recognition (AVSR)
• scripts/ ← Shell and Python scripts to orchestrate preprocessing, inference, training
• main.py & train_*.sh ← Entrypoints for inference & training
• Models & Modules under sgm/:
• sgm/models/ ← Neural network definitions (autoencoder, diffusion)
• sgm/modules/ ← Building blocks (attention, temporal AE, loss, sampling)
• sgm/data/ ← PyTorch DataModules and datasets
• sgm/inference/ ← Inference API & helpers
• evaluation/auto_avsr/: separate AVSR benchmark to evaluate lip-sync quality
• notebooks/: exploratory analysis
Key Components
a. Data Ingestion & Preprocessing
• Video inputs → Face landmark detection (Mediapipe/Retinaface) → Video embeddings via autoencoder
• Audio inputs → Feature extraction (WavLM, HuBERT) → Audio embeddings
b. Two-Stage Models
1. Keyframe Model (sgm/models/autoencoder & sgm/modules/autoencoding)
• Learns to predict masked lip keyframes
2. Interpolation Model (sgm/models/diffusion & sgm/modules/diffusionmodules)
• Denoising diffusion to interpolate smooth lip motion between keyframes
c. Orchestration Layers
• PyTorch Lightning trainers (sgm/callbacks & lightning.py)
• Configuration management via OmegaConf (configs/*.yaml)
• Shell scripts for end-to-end pipelines (scripts/infer_raw.sh, train_keyframe.sh, train_interpolation.sh)
d. Evaluation & Metrics
• LipScore module (evaluation/lipscore.py)
• Optional AVSR evaluation (evaluation/auto_avsr/)
Relationships and Data Flow
• Video embeddings and audio embeddings converge into Keyframe stage → output keyframe predictions
• Keyframe outputs feed into Interpolation stage → final video reconstruction
• The reconstruction method invokes the trained autoencoder decoder to produce pixel frames
• Evaluation components take generated video + ground truth video as inputs → LipScore and AVSR metrics
Architectural Patterns & Design Principles
• Modular ML pipeline: clear separation of data, model, and inference code
• Two-stage “coarse-to-fine” framework (keyframe + diffusion)
• Configurable via Hydra/OmegaConf for reproducibility
• PyTorch Lightning for standardized training loops and callbacks
• Extensibility: separate sgm/modules for plugging new loss functions or diffusion strategies
Essential Elements Checklist
[ ] Clear labels for each component
[ ] References to config files triggering each stage