# jixiaozhong/sonic architecture

> Architecture diagram of the GitHub repository jixiaozhong/sonic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jixiaozhong/sonic
- Repository: https://github.com/jixiaozhong/sonic

## Overview

##### Project type and purpose
- Sonic is a research‐grade inference pipeline for audio-driven portrait animation (CVPR 2025).
- It is not a multi-service web app but a modular ML inference system with both CLI and Gradio-based frontends, a preprocessing step, a core diffusion-based animation pipeline, and post-processing components.

##### File structure
- Top‐level scripts:
- demo.py / demo.sh – command-line entry point
- gradio_app.py – interactive web UI via Gradio
- config/inference – YAML configs for models and hyperparameters
- src/:
- dataset/face_align – face detection & alignment (yoloface, align.py)
- models/
– audio_adapter (audio_proj, audio_to_bucket)
– base (attention processors, UNet 3D blocks, spatio-temporal conditioning)
- pipelines/pipeline_sonic.py – orchestrates the core diffusion pipeline
- utils/
– RIFE frame interpolation
– mask_processor for compositing
– generic utilities
- examples/, requirements.txt, checkpoints/ (external model storage)

##### Key insights from README
- Core technologies: PyTorch, Huggingface model loading, stable-video-diffusion-img2vid-xt, Whisper, RIFE, YOLOFace.
- Inference workflow:
1. Load input image + audio
2. Face alignment
3. Audio feature extraction & bucketing
4. Video diffusion (UNet3D) with spatio-temporal conditioning
5. Frame interpolation & masking
6. Output video
- Models are downloaded via Huggingface Hub into checkpoints/.

##### Main components
a. Frontend layer
- CLI (demo.py/demo.sh)
- Gradio web UI (gradio_app.py)
b. Preprocessing
- Face detector & aligner (src/dataset/face_align)
- Audio processor (src/models/audio_adapter: audio_to_bucket, audio_proj)
c. Core pipeline
- pipeline_sonic.py orchestrator
- Base modules (attention_processor, UNet_3D_Blocks, spatio-temporal conditioning)
- Diffusion engine (stable-video-diffusion-img2vid-xt)
d. Post-processing
- Mask processor (src/utils/mask_processor.py)
- Frame interpolation (RIFE: utils/RIFE/*)
e. Model & config storage
- checkpoints/ (Huggingface Hub downloads)
- config/inference/sonic.yaml
f. Output writer
- Video encoder (e.g., ffmpeg wrapper in demo or pipeline)

##### Relationships and interactions
- Frontend → Preprocessing: UI passes image/audio paths to preprocessing modules.
- Preprocessing → Core pipeline: Aligned face image + audio features → pipeline_sonic.
- Core pipeline loads model weights from checkpoints (via Huggingface Hub API).
- Core pipeline → Post-processing: Raw diffusion frames → mask & RIFE interpolation.
- Post-processing → Output writer: Final video file.
- Config/inference → all model modules for hyperparams and paths.

##### Architectural patterns & principles
- Modular pipeline pattern: clear separation of stages (preprocess/core/postprocess).
- Dependency injection via YAML config.
- Pluggable adapters (audio_adapter, base UNet blocks).
- Reusable utilities (RIFE, mask processor) abstracted under utils/.

##### Relevant technologies & frameworks
- Python 3, PyTorch, CUDA
- Huggingface Hub CLI & API
- Gradio for web inference
- YOLOFace for detection, Whisper for audio (external models)
- RIFE for frame interpolation
- YAML for configuration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        CLI["CLI Entry (demo.py/demo.sh)"]:::compute
        Gradio["Gradio UI (gradio_app.py)"]:::compute
    end

    %% Preprocessing Stage
    subgraph "Preprocessing" 
        Yolo["YoloFace Detector"]:::compute
        Align["Face Aligner (align.py)"]:::compute
        AudioProj["Audio Projection"]:::compute
        AudioBucket["Audio Bucketing"]:::compute
    end

    %% Core Pipeline Stage
    subgraph "Core Pipeline"
        Orchestrator["Pipeline Sonic Orchestrator"]:::compute
        Attention["Attention Processor"]:::compute
        UNet3D["UNet 3D Blocks"]:::compute
        Spatio["Spatio-Temporal Conditioning"]:::compute
        Diffusion["Diffusion Engine\n(stable-video-diffusion-img2vid-xt)"]:::compute
    end

    %% Post-Processing Stage
    subgraph "Post-Processing"
        MaskProc["Mask Processor"]:::compute
        IFNet["RIFE IFNet_HDv3"]:::compute
        RIFEHD["RIFE_HDv3"]:::compute
        Warplayer["RIFE Warplayer"]:::compute
    end

    %% Storage & Config
    subgraph "Storage & Config"
        Checkpoints[(Checkpoints)]:::storage
        Config["sonic.yaml"]:::config
    end

    %% External Dependencies
    subgraph "External Services"
        HFHub["Huggingface Hub"]:::external
        GPU["GPU (CUDA)"]:::external
    end

    %% Output Writer
    VideoEnc["Video Encoder (ffmpeg wrapper)"]:::compute
    OutputFile["Final Video File"]:::data

    %% Data Flows
    CLI -->|"image+audio paths"| Yolo
    CLI -->|"image+audio paths"| AudioProj
    Gradio -->|"image+audio paths"| Yolo
    Gradio -->|"image+audio paths"| AudioProj

    Yolo -->|"aligned face bbox"| Align
    Align -->|"aligned face image"| Orchestrator

    AudioProj -->|"audio embeddings"| AudioBucket
    AudioBucket -->|"audio feature buckets"| Orchestrator

    Orchestrator -->|"calls attention"| Attention
    Orchestrator -->|"calls UNet3D"| UNet3D
    Orchestrator -->|"applies conditioning"| Spatio
    Orchestrator -->|"generates frames"| Diffusion

    Checkpoints -->|"model weights"| Orchestrator
    Config -->|"hyperparams/load paths"| Orchestrator

    Orchestrator -->|"raw frames"| MaskProc
    MaskProc -->|"masked frames"| IFNet
    IFNet -->|"interpolated frames"| RIFEHD
    RIFEHD -->|"warped frames"| Warplayer
    Warplayer -->|"final frames"| VideoEnc

    VideoEnc -->|"encoded video"| OutputFile

    HFHub -->|"downloads models"| Checkpoints
    GPU -->|"compute acceleration"| Orchestrator

    %% Click Events
    click CLI "https://github.com/jixiaozhong/sonic/blob/main/demo.py"
    click CLI "https://github.com/jixiaozhong/sonic/blob/main/demo.sh"
    click Gradio "https://github.com/jixiaozhong/sonic/blob/main/gradio_app.py"
    click Config "https://github.com/jixiaozhong/sonic/blob/main/config/inference/sonic.yaml"
    click Yolo "https://github.com/jixiaozhong/sonic/blob/main/src/dataset/face_align/yoloface.py"
    click Align "https://github.com/jixiaozhong/sonic/blob/main/src/dataset/face_align/align.py"
    click AudioProj "https://github.com/jixiaozhong/sonic/blob/main/src/models/audio_adapter/audio_proj.py"
    click AudioBucket "https://github.com/jixiaozhong/sonic/blob/main/src/models/audio_adapter/audio_to_bucket.py"
    click Orchestrator "https://github.com/jixiaozhong/sonic/blob/main/src/pipelines/pipeline_sonic.py"
    click Attention "https://github.com/jixiaozhong/sonic/blob/main/src/models/base/attention_processor.py"
    click UNet3D "https://github.com/jixiaozhong/sonic/blob/main/src/models/base/unet_3d_blocks.py"
    click Spatio "https://github.com/jixiaozhong/sonic/blob/main/src/models/base/unet_spatio_temporal_condition.py"
    click MaskProc "https://github.com/jixiaozhong/sonic/blob/main/src/utils/mask_processer.py"
    click IFNet "https://github.com/jixiaozhong/sonic/blob/main/src/utils/RIFE/IFNet_HDv3.py"
    click RIFEHD "https://github.com/jixiaozhong/sonic/blob/main/src/utils/RIFE/RIFE_HDv3.py"
    click Warplayer "https://github.com/jixiaozhong/sonic/blob/main/src/utils/RIFE/warplayer.py"
    click VideoEnc "https://github.com/jixiaozhong/sonic/blob/main/src/utils/util.py"

    %% Styles
    classDef data fill:#bbdefb,stroke:#333,stroke-width:1px
    classDef compute fill:#c8e6c9,stroke:#333,stroke-width:1px
    classDef storage fill:#ffe0b2,stroke:#333,stroke-width:1px
    classDef external fill:#e1bee7,stroke:#333,stroke-width:1px
    classDef config fill:#ffcc80,stroke:#333,stroke-width:1px
```
