How bytedance/latentsync works
Project Type and Purpose
• LatentSync is an open-source, end-to-end audio-conditioned latent diffusion lip-sync tool and research pipeline.
• It consists of a data processing pipeline, model training (SyncNet & U-Net latent diffusion), evaluation scripts, and inference modes (CLI or Gradio UI).
File Structure Analysis
Top-level folders and what they signal:
• preprocess/ & data_processing_pipeline.sh – audio/video cleaning & segmentation pipeline
• latentsync/ – core model code (data loaders, Whisper audio encoder, U-Net diffusion, TREPA losses, SyncNet supervision)
• configs/ – YAML/JSON configs for SyncNet, U-Net training stages
• eval/ – evaluation scripts for sync confidence, FVD, hyperIQA
• scripts/ & train_*.sh – orchestration for training SyncNet/U-Net
• gradio_app.py & inference.sh – inference entry points
• tools/ – utility scripts (file listing, video metrics, downloading, GPU management)
Key Architecture Insights from README
• Audio → Whisper encoder → audio embeddings
• Latent diffusion U-Net with cross-attention integrating audio embeddings + reference/masked frames
• One-step denoising → decode latents → reconstructed frames
• Supervision via TREPA, LPIPS, SyncNet losses in pixel space
• Two main workflows: Training (data → preprocess → train SyncNet/U-Net) and Inference (input video/audio → generate lip-synced output)