# bytedance/latentsync architecture

> Architecture diagram of the GitHub repository bytedance/latentsync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bytedance/latentsync
- Repository: https://github.com/bytedance/latentsync

## Overview

##### Project Type and Purpose
- LatentSync is an open-source, end-to-end audio-conditioned latent diffusion lip-sync tool and research pipeline.
- It consists of a data processing pipeline, model training (SyncNet & U-Net latent diffusion), evaluation scripts, and inference modes (CLI or Gradio UI).

##### File Structure Analysis
Top-level folders and what they signal:
- preprocess/ & data_processing_pipeline.sh – audio/video cleaning & segmentation pipeline
- latentsync/ – core model code (data loaders, Whisper audio encoder, U-Net diffusion, TREPA losses, SyncNet supervision)
- configs/ – YAML/JSON configs for SyncNet, U-Net training stages
- eval/ – evaluation scripts for sync confidence, FVD, hyperIQA
- scripts/ & train_*.sh – orchestration for training SyncNet/U-Net
- gradio_app.py & inference.sh – inference entry points
- tools/ – utility scripts (file listing, video metrics, downloading, GPU management)

##### Key Architecture Insights from README
- Audio → Whisper encoder → audio embeddings
- Latent diffusion U-Net with cross-attention integrating audio embeddings + reference/masked frames
- One-step denoising → decode latents → reconstructed frames
- Supervision via TREPA, LPIPS, SyncNet losses in pixel space
- Two main workflows: Training (data → preprocess → train SyncNet/U-Net) and Inference (input video/audio → generate lip-synced output)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top-level Data Storage
    RawAssets[(Raw Assets & Data\nassets/)]:::utility
    click RawAssets "https://github.com/bytedance/latentsync/tree/main/assets/"

    %% Data Processing Pipeline
    subgraph "Data Processing Pipeline"
        direction TB
        ProcessScript["data_processing_pipeline.sh"]:::preprocess
        click ProcessScript "https://github.com/bytedance/latentsync/blob/main/data_processing_pipeline.sh"
        PreprocessModule["preprocess/"]:::preprocess
        click PreprocessModule "https://github.com/bytedance/latentsync/tree/main/preprocess/"
    end

    RawAssets --> ProcessScript
    RawAssets --> PreprocessModule

    %% Model Training
    subgraph "Model Training" 
        direction TB

        subgraph "SyncNet Training" 
            direction TB
            SNConfig["configs/syncnet/"]:::training
            click SNConfig "https://github.com/bytedance/latentsync/tree/main/configs/syncnet/"
            SNTrainScript["train_syncnet.sh & scripts/train_syncnet.py"]:::training
            click SNTrainScript "https://github.com/bytedance/latentsync/blob/main/train_syncnet.sh"
            click SNTrainScript "https://github.com/bytedance/latentsync/blob/main/scripts/train_syncnet.py"
            SNModel["latentsync/models/stable_syncnet.py"]:::training
            click SNModel "https://github.com/bytedance/latentsync/blob/main/latentsync/models/stable_syncnet.py"
        end

        subgraph "U-Net Latent Diffusion Training"
            direction TB
            UNetConfig["configs/unet/"]:::training
            click UNetConfig "https://github.com/bytedance/latentsync/tree/main/configs/unet/"
            UNETScript["train_unet.sh & scripts/train_unet.py"]:::training
            click UNETScript "https://github.com/bytedance/latentsync/blob/main/train_unet.sh"
            click UNETScript "https://github.com/bytedance/latentsync/blob/main/scripts/train_unet.py"
            Whisper["latentsync/whisper/audio2feature.py"]:::training
            click Whisper "https://github.com/bytedance/latentsync/blob/main/latentsync/whisper/audio2feature.py"
            UNetModel1["latentsync/models/unet.py"]:::training
            click UNetModel1 "https://github.com/bytedance/latentsync/blob/main/latentsync/models/unet.py"
            UNetModel2["latentsync/models/unet_blocks.py"]:::training
            click UNetModel2 "https://github.com/bytedance/latentsync/blob/main/latentsync/models/unet_blocks.py"
            TREPA["latentsync/trepa/loss.py"]:::training
            click TREPA "https://github.com/bytedance/latentsync/blob/main/latentsync/trepa/loss.py"
        end

        %% Data flow to trainings
        ProcessScript --> SNTrainScript
        PreprocessModule --> SNTrainScript
        ProcessScript --> UNETScript
        PreprocessModule --> UNETScript
    end

    %% Evaluation
    subgraph "Evaluation" 
        direction TB
        Eval1["eval/eval_sync_conf.sh"]:::evaluation
        click Eval1 "https://github.com/bytedance/latentsync/blob/main/eval/eval_sync_conf.sh"
        Eval2["eval/eval_syncnet_acc.sh"]:::evaluation
        click Eval2 "https://github.com/bytedance/latentsync/blob/main/eval/eval_syncnet_acc.sh"
        Eval3["eval/eval_fvd.py"]:::evaluation
        click Eval3 "https://github.com/bytedance/latentsync/blob/main/eval/eval_fvd.py"
        Eval4["eval/hyper_iqa.py"]:::evaluation
        click Eval4 "https://github.com/bytedance/latentsync/blob/main/eval/hyper_iqa.py"
        EvalAll["eval/ (all scripts)"]:::evaluation
        click EvalAll "https://github.com/bytedance/latentsync/tree/main/eval/"
    end

    SNModel --> Eval1
    SNModel --> Eval2
    UNetModel1 --> Eval3
    UNetModel1 --> Eval4
    UNetModel1 --> EvalAll

    %% Inference
    subgraph "Inference" 
        direction TB
        CLI["CLI Mode"]:::inference
        click CLI "https://github.com/bytedance/latentsync/blob/main/inference.sh"
        click CLI "https://github.com/bytedance/latentsync/blob/main/scripts/inference.py"
        click CLI "https://github.com/bytedance/latentsync/blob/main/predict.py"
        Gradio["Gradio UI"]:::inference
        click Gradio "https://github.com/bytedance/latentsync/blob/main/gradio_app.py"
        CLI --> Gradio
    end

    UNetModel1 --> CLI
    SNModel --> CLI
    UNetModel1 --> Gradio
    SNModel --> Gradio

    %% Utilities
    Tools["tools/"]:::utility
    click Tools "https://github.com/bytedance/latentsync/tree/main/tools/"

    Tools --> ProcessScript
    Tools --> SNTrainScript
    Tools --> UNETScript
    Tools --> EvalAll
    Tools --> CLI

    %% Styles
    classDef preprocess fill:#ADD8E6,stroke:#000;
    classDef training fill:#90EE90,stroke:#000;
    classDef evaluation fill:#FFA500,stroke:#000;
    classDef inference fill:#DDA0DD,stroke:#000;
    classDef utility fill:#D3D3D3,stroke:#000;
```
