How jixiaozhong/sonic works
Project type and purpose
• Sonic is a research‐grade inference pipeline for audio-driven portrait animation (CVPR 2025).
• It is not a multi-service web app but a modular ML inference system with both CLI and Gradio-based frontends, a preprocessing step, a core diffusion-based animation pipeline, and post-processing components.
File structure
• Top‐level scripts:
• demo.py / demo.sh – command-line entry point
• gradio_app.py – interactive web UI via Gradio
• config/inference – YAML configs for models and hyperparameters
• src/:
• dataset/face_align – face detection & alignment (yoloface, align.py)
• models/
– audio_adapter (audio_proj, audio_to_bucket)
– base (attention processors, UNet 3D blocks, spatio-temporal conditioning)
• pipelines/pipeline_sonic.py – orchestrates the core diffusion pipeline
• utils/
– RIFE frame interpolation
– mask_processor for compositing
– generic utilities
• examples/, requirements.txt, checkpoints/ (external model storage)
Key insights from README
• Core technologies: PyTorch, Huggingface model loading, stable-video-diffusion-img2vid-xt, Whisper, RIFE, YOLOFace.
• Inference workflow:
1. Load input image + audio
2. Face alignment
3. Audio feature extraction & bucketing
4. Video diffusion (UNet3D) with spatio-temporal conditioning
5. Frame interpolation & masking
6. Output video
• Models are downloaded via Huggingface Hub into checkpoints/.
Main components
a. Frontend layer
• CLI (demo.py/demo.sh)
• Gradio web UI (gradio_app.py)
b. Preprocessing
• Face detector & aligner (src/dataset/face_align)
• Audio processor (src/models/audio_adapter: audio_to_bucket, audio_proj)
c. Core pipeline
• pipeline_sonic.py orchestrator
• Base modules (attention_processor, UNet_3D_Blocks, spatio-temporal conditioning)
• Diffusion engine (stable-video-diffusion-img2vid-xt)
d. Post-processing
• Mask processor (src/utils/mask_processor.py)
• Frame interpolation (RIFE: utils/RIFE/*)
e. Model & config storage
• checkpoints/ (Huggingface Hub downloads)
• config/inference/sonic.yaml
f. Output writer
• Video encoder (e.g., ffmpeg wrapper in demo or pipeline)
Relationships and interactions
• Frontend → Preprocessing: UI passes image/audio paths to preprocessing modules.
• Preprocessing → Core pipeline: Aligned face image + audio features → pipeline_sonic.
• Core pipeline loads model weights from checkpoints (via Huggingface Hub API).
• Core pipeline → Post-processing: Raw diffusion frames → mask & RIFE interpolation.
• Post-processing → Output writer: Final video file.
• Config/inference → all model modules for hyperparams and paths.
Architectural patterns & principles
• Modular pipeline pattern: clear separation of stages (preprocess/core/postprocess).
• Dependency injection via YAML config.
• Pluggable adapters (audio_adapter, base UNet blocks).
• Reusable utilities (RIFE, mask processor) abstracted under utils/.
Relevant technologies & frameworks
• Python 3, PyTorch, CUDA
• Huggingface Hub CLI & API
• Gradio for web inference
• YOLOFace for detection, Whisper for audio (external models)
• RIFE for frame interpolation
• YAML for configuration