How ali-vilab/unianimate-dit works
Project Type & Purpose
• Type: An open-source video diffusion toolkit for consistent human image animation (full training + inference).
• Purpose: Provide CLI scripts, UI apps (Gradio/Streamlit), and a Python library (diffsynth) that implements data preprocessing, pose alignment, prompt engineering, diffusion pipelines, extensions (ESRGAN, FastBlend, RIFE, IQMetric), and training via PEFT/Lightning/DeepSpeed.
Main Components
A. Frontend/UI Layer
• apps/gradio/DiffSynth_Studio.py
• apps/streamlit/DiffSynth_Studio.py + pages (Image Creator, Video Creator)
• Purpose: User interacts via web GUI to upload reference image/video, configure parameters, and trigger inference.
B. Data & Pose Alignment
• run_align_pose.py
• dwpose module (onnxdet.py, onnxpose.py, wholebody.py)
• prepare_training_data.py + data/example_dataset
• Purpose: Extract and align pose sequences for reference images and source videos; package frames for training.
C. Core Library (diffsynth package)
1. configs & tokenizer_configs: Model/hyperparameter definitions, tokenizer assets.
2. models: DiT/U-Net architectures, text encoders, VAE encoders/decoders, LoRA modules, model manager.
3. pipelines: PipelineRunner + task-specific pipelines (WanVideoPipeline, WanUniAnimateVideoPipeline, sd3_image, step_video, etc.).
4. prompters: Generate conditioning text prompts for various backbone models.
5. schedulers: Diffusion schedulers (DDIM, ODE, FlowMatch).
6. controlnets: Plug-in conditioning networks for pose, depth, etc.
7. processors: Post-processing (FastBlend, PILEditor, RIFE).
8. extensions:
• ImageQualityMetric (BLIP, CLIP, Aesthetic, etc.)
• ESRGAN (super-resolution)
• FastBlend (image/video blending)
• RIFE (frame interpolation)
9. distributed: xdit_context_parallel for multi-GPU Unified Sequence Parallel (USP).
10. trainers: text_to_image training scripts via PEFT/Lightning/DeepSpeed.
11. vram_management: Layers to optimize GPU memory usage.
D. Examples & Scripts
• examples/unianimate_wan: inference and training entry-points for UniAnimate-DiT.
• examples/wanvideo: vanilla Wan2.1 image/video pipelines.
• setup.py, requirements.txt, huggingface/modelscope CLIs.
Relationships & Interactions
• Frontend → run_align_pose (aligned pose output) → Core PipelineRunner
• PipelineRunner loads:
1. ModelManager (VAE, Text Encoder, DiT/U-Net weights, LoRA adapters)
2. Tokenizers (from tokenizer_configs)
• Pipeline for inference:
– Input: reference image + pose sequence + text prompt
– Prompter → Tokenizer → Conditioning embeddings
– Scheduler request timesteps → DiT/U-Net
– ControlNets inject pose guidance
– Post-processors chain (FastBlend, RIFE, ESRGAN, IQMetric)
– Output: video frames → save_video
• Training pipeline:
– DataLoader (packaged poses + frames) → VAE feature extraction → Diffusion training with LoRA
– LightningModule / DeepSpeed for multi-GPU.
– Checkpoint outputs (.ckpt / .safetensors).
Architectural Patterns & Principles
• Modular “plugin” architecture: Each extension (controlnet, processor, IQMetric) is a self-contained module.
• Pipeline pattern: Data flows through ordered stages (prompter → model → post-processor).
• Separation of concerns: UI layer, data preprocessing, core library, training, and examples are cleanly separated.
• Scalable distributed execution via USP and xdit_context_parallel.
• Use of PEFT to decouple base model weights from fine-tuned LoRA adapters.