# ali-vilab/unianimate-dit architecture

> Architecture diagram of the GitHub repository ali-vilab/unianimate-dit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ali-vilab/unianimate-dit
- Repository: https://github.com/ali-vilab/unianimate-dit

## Overview

##### Project Type & Purpose
- Type: An open-source video diffusion toolkit for consistent human image animation (full training + inference).
- Purpose: Provide CLI scripts, UI apps (Gradio/Streamlit), and a Python library (diffsynth) that implements data preprocessing, pose alignment, prompt engineering, diffusion pipelines, extensions (ESRGAN, FastBlend, RIFE, IQMetric), and training via PEFT/Lightning/DeepSpeed.

##### Main Components
A. Frontend/UI Layer
- apps/gradio/DiffSynth_Studio.py
- apps/streamlit/DiffSynth_Studio.py + pages (Image Creator, Video Creator)
- Purpose: User interacts via web GUI to upload reference image/video, configure parameters, and trigger inference.
B. Data & Pose Alignment
- run_align_pose.py
- dwpose module (onnxdet.py, onnxpose.py, wholebody.py)
- prepare_training_data.py + data/example_dataset
- Purpose: Extract and align pose sequences for reference images and source videos; package frames for training.
C. Core Library (diffsynth package)
1. configs & tokenizer_configs: Model/hyperparameter definitions, tokenizer assets.
2. models: DiT/U-Net architectures, text encoders, VAE encoders/decoders, LoRA modules, model manager.
3. pipelines: PipelineRunner + task-specific pipelines (WanVideoPipeline, WanUniAnimateVideoPipeline, sd3_image, step_video, etc.).
4. prompters: Generate conditioning text prompts for various backbone models.
5. schedulers: Diffusion schedulers (DDIM, ODE, FlowMatch).
6. controlnets: Plug-in conditioning networks for pose, depth, etc.
7. processors: Post-processing (FastBlend, PILEditor, RIFE).
8. extensions:
- ImageQualityMetric (BLIP, CLIP, Aesthetic, etc.)
- ESRGAN (super-resolution)
- FastBlend (image/video blending)
- RIFE (frame interpolation)
9. distributed: xdit_context_parallel for multi-GPU Unified Sequence Parallel (USP).
10. trainers: text_to_image training scripts via PEFT/Lightning/DeepSpeed.
11. vram_management: Layers to optimize GPU memory usage.
D. Examples & Scripts
- examples/unianimate_wan: inference and training entry-points for UniAnimate-DiT.
- examples/wanvideo: vanilla Wan2.1 image/video pipelines.
- setup.py, requirements.txt, huggingface/modelscope CLIs.

##### Relationships & Interactions
- Frontend → run_align_pose (aligned pose output) → Core PipelineRunner
- PipelineRunner loads:
1. ModelManager (VAE, Text Encoder, DiT/U-Net weights, LoRA adapters)
2. Tokenizers (from tokenizer_configs)
- Pipeline for inference:
– Input: reference image + pose sequence + text prompt
– Prompter → Tokenizer → Conditioning embeddings
– Scheduler request timesteps → DiT/U-Net
– ControlNets inject pose guidance
– Post-processors chain (FastBlend, RIFE, ESRGAN, IQMetric)
– Output: video frames → save_video
- Training pipeline:
– DataLoader (packaged poses + frames) → VAE feature extraction → Diffusion training with LoRA
– LightningModule / DeepSpeed for multi-GPU.
– Checkpoint outputs (.ckpt / .safetensors).

##### Architectural Patterns & Principles
- Modular “plugin” architecture: Each extension (controlnet, processor, IQMetric) is a self-contained module.
- Pipeline pattern: Data flows through ordered stages (prompter → model → post-processor).
- Separation of concerns: UI layer, data preprocessing, core library, training, and examples are cleanly separated.
- Scalable distributed execution via USP and xdit_context_parallel.
- Use of PEFT to decouple base model weights from fine-tuned LoRA adapters.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Frontend/UI Layer"
        GradioApp["Gradio App"]:::frontend
        StreamlitApp["Streamlit App"]:::frontend
        ImgPage["Image Creator Page"]:::frontend
        VidPage["Video Creator Page"]:::frontend
    end

    subgraph "Data & Preprocessing"
        AlignScript["Pose Extraction"]:::pre
        PoseUtils["Pose Utils"]:::pre
        PrepData["Prepare Training Data"]:::pre
        DataAssets["Example Dataset"]:::data
    end

    subgraph "DiffSynth Core Library"
        subgraph "Configurations"
            ModelConfig["Model Config"]:::core
            TokenizerConfig["Tokenizer Configs"]:::core
        end
        subgraph "Models"
            ModelMgr["Model Manager"]:::core
            ModelDefs["Model Definitions"]:::core
        end
        subgraph "Pipelines"
            PipelineRunner["Pipeline Runner"]:::core
            SubPipelines["Task Pipelines"]:::core
        end
        Prompters["Prompters"]:::core
        Schedulers["Schedulers"]:::core
        ControlNets["ControlNets"]:::core
        Processors["Processors"]:::core
        subgraph "Extensions"
            ESRGAN["ESRGAN"]:::ext
            FastBlend["FastBlend"]:::ext
            IQMetric["Image Quality Metrics"]:::ext
            RIFEExt["RIFE"]:::ext
        end
        Distributed["Distributed Exec"]:::core
        Trainers["Training Utilities"]:::core
        VRAM["VRAM Management"]:::core
    end

    subgraph "GPU Runtime"
        GPU["GPU Cluster"]:::gpu
    end

    subgraph "Examples & Scripts"
        UniEntry["UniAnimate Entry Points"]:::example
        WanEntry["WanVideo Entry Points"]:::example
        Setup["Setup & Dependencies"]:::example
    end

    subgraph "External Services"
        HF["HuggingFace Hub"]:::external
        ORT["ONNX Runtime"]:::external
        DS["DeepSpeed/Lightning"]:::external
    end

    GradioApp -->|"invoke"| AlignScript
    StreamlitApp -->|"invoke"| AlignScript
    AlignScript -->|"extract poses"| PoseUtils
    PoseUtils -->|"aligned poses"| PrepData
    PrepData -->|"data package"| DataAssets

    DataAssets -->|"load"| PipelineRunner
    AlignScript -->|"poses"| PipelineRunner

    PipelineRunner -->|"load configs"| ModelConfig
    PipelineRunner -->|"load tokenizers"| TokenizerConfig
    PipelineRunner -->|"load models"| ModelMgr
    ModelMgr -->|"weights"| SubPipelines

    Prompters -->|"generate cond"| SubPipelines
    Schedulers -->|"timesteps"| SubPipelines
    ControlNets -->|"guidance"| SubPipelines

    SubPipelines -->|"inference"| Processors
    Processors -->|"post-process"| GPU

    SubPipelines -->|"train"| VRAM
    VRAM -->|"optimize"| GPU
    Trainers -->|"train"| GPU
    Distributed -->|"parallel"| GPU

    GPU -->|"outputs"| UniEntry
    GPU -->|"outputs"| WanEntry
    UniEntry -->|"packages"| Setup
    WanEntry -->|"packages"| Setup

    HF --> PipelineRunner
    ORT --> PoseUtils
    DS --> Trainers

    click GradioApp "https://github.com/ali-vilab/unianimate-dit/blob/main/apps/gradio/DiffSynth_Studio.py"
    click StreamlitApp "https://github.com/ali-vilab/unianimate-dit/blob/main/apps/streamlit/DiffSynth_Studio.py"
    click ImgPage "https://github.com/ali-vilab/unianimate-dit/blob/main/apps/streamlit/pages/1_Image_Creator.py"
    click VidPage "https://github.com/ali-vilab/unianimate-dit/blob/main/apps/streamlit/pages/2_Video_Creator.py"
    click AlignScript "https://github.com/ali-vilab/unianimate-dit/blob/main/run_align_pose.py"
    click PoseUtils "https://github.com/ali-vilab/unianimate-dit/blob/main/dwpose/onnxdet.py"
    click PoseUtils "https://github.com/ali-vilab/unianimate-dit/blob/main/dwpose/onnxpose.py"
    click PoseUtils "https://github.com/ali-vilab/unianimate-dit/blob/main/dwpose/wholebody.py"
    click PrepData "https://github.com/ali-vilab/unianimate-dit/blob/main/prepare_training_data.py"
    click DataAssets "https://github.com/ali-vilab/unianimate-dit/tree/main/data/example_dataset"
    click ModelConfig "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/configs/model_config.py"
    click TokenizerConfig "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/tokenizer_configs"
    click ModelMgr "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/models/model_manager.py"
    click ModelDefs "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/models/"
    click PipelineRunner "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/pipelines/pipeline_runner.py"
    click SubPipelines "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/pipelines/sd_image.py"
    click SubPipelines "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/pipelines/sd_video.py"
    click SubPipelines "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/pipelines/wan_video.py"
    click SubPipelines "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/pipelines/step_video.py"
    click Prompters "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/prompters/"
    click Schedulers "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/schedulers/"
    click ControlNets "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/controlnets/"
    click Processors "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/processors/"
    click ESRGAN "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/extensions/ESRGAN/"
    click FastBlend "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/extensions/FastBlend/"
    click IQMetric "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/extensions/ImageQualityMetric/"
    click RIFEExt "https://github.com/ali-vilab/unianimate-dit/tree/main/diffsynth/extensions/RIFE/"
    click Distributed "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/distributed/xdit_context_parallel.py"
    click Trainers "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/trainers/text_to_image.py"
    click VRAM "https://github.com/ali-vilab/unianimate-dit/blob/main/diffsynth/vram_management/layers.py"
    click UniEntry "https://github.com/ali-vilab/unianimate-dit/tree/main/examples/unianimate_wan/"
    click WanEntry "https://github.com/ali-vilab/unianimate-dit/tree/main/examples/wanvideo/"
    click Setup "https://github.com/ali-vilab/unianimate-dit/blob/main/setup.py"

    classDef frontend fill:#cef,stroke:#036,stroke-width:1px
    classDef pre fill:#cfc,stroke:#090,stroke-width:1px
    classDef data fill:#afa,stroke:#070,stroke-width:1px
    classDef core fill:#fcb,stroke:#f80,stroke-width:1px
    classDef ext fill:#fcc,stroke:#f00,stroke-width:1px
    classDef gpu fill:#eec,stroke:#c60,stroke-width:1px
    classDef example fill:#ddf,stroke:#44f,stroke-width:1px
    classDef external fill:#eee,stroke:#888,stroke-width:1px
```
