# fudan-generative-vision/champ architecture

> Architecture diagram of the GitHub repository fudan-generative-vision/champ, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fudan-generative-vision/champ
- Repository: https://github.com/fudan-generative-vision/champ

## Overview

##### Project Type and Purpose
- This project is a research-oriented deep learning framework for human image animation using 3D parametric guidance. It is not a typical full-stack application but rather a pipeline that involves data processing, model building, training, and inference.
- The README, along with the file tree, shows two main workflows: inference (animation generation) and training (with its two stages).

##### File Structure to Identify Main Components
- Configuration and Workflow:
- The “configs” directory (with separate configs for training [stage1.yaml, stage2.yaml] and inference [inference.yaml]) drives the behavior of both training and inference.
- Data Modules and Preprocessing:
- “datasets” contains dataset handling scripts (video, image datasets, data utilities).
- “scripts/data_processors” includes specialized SMPL and DWPose processing scripts (for blending, generating SMPL data and visualization).
- Model Modules:
- “models” contains various network architectures such as attention, transformers (2D and 3D), UNet blocks, motion modules, etc.
- Pipeline Components:
- “pipelines” provides functions for aggregating contexts and bridging between guidance inputs and final outputs (e.g., pipeline_guidance2image, pipeline_guidance2video).
- Entry Points:
- “inference.py” is the entrypoint for the inference pipeline.
- “train_s1.py” and “train_s2.py” are separate scripts for the two-stage training process.
- Utility and Support:
- “utils” contains file system helpers, tracker for TensorBoard, and video processing utilities.
- “docs”, “assets”, and “images” provide documentation, visual assets, and Docker/containers for deployment support.

- Data Processing & Preprocessing:
- Data sources (training videos, motion samples from external preparation such as SMPL & DWPose).
- Dedicated data processor scripts (e.g., generate_dwpose, generate_smpls, smpl_transfer).
- Configuration Layer:
- Config files for inference and training that guide behavior.
- Model Components:
- Different model blocks (attention net, transformer layers, UNet-based modules, motion module, guidance encoders) from the “models” directory.
- Pipeline Components:
- Components from “pipelines” that combine input data and model outputs to generate final animation.
- Training and Inference Entrypoints:
- “train_s1.py” and “train_s2.py” for training stages.
- “inference.py” for running inference, including handling of pretrained models, guidance data, and result generation.
- External Dependencies:
- Pretrained models (downloaded from HuggingFace and organized under “pretrained_models”).
- External frameworks like Conda, Poetry, Accelerate, and underlying libraries (PyTorch, diffusers, etc.).

b. Relationships and Data Flow:
- Data Flow: Training data (video frames, depth, dwpose, mask, normal, semantic maps) and guidance motion data flow into the preprocessing pipelines (located in scripts and datasets).
- The configuration files define parameters for both training (stage1 and stage2) and inference which then control how data is fed into the pipelines and subsequently into the models.
- After processing, the data enters the model components, feeding into training routines (train_s1.py → train_s2.py) and inference (inference.py).
- Outputs flow from the model and pipelines towards generating animations (stored in a “results” folder) or updating pretrained model checkpoints.

##### Architectural Patterns
- Modularity: The project is clearly divided into modules (datasets, models, pipelines, utils, scripts) which allows separate understanding and development of each component.
- Two-Stage Training Pipeline: The training process is separated into two stages, with different configs and entrypoints allowing for progressive refinement.
- Data Flow Driven via YAML Configurations: Centralized configuration files provide flexibility and reduce hard-coded dependencies across components.

##### Final Touches
- Keep the design clean yet comprehensive: major blocks/components should be visible along with interconnections, so that stakeholders and engineers can understand how each module contributes to the whole system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data
    A["Raw Data (Video/Motion Samples)"]:::external

    %% Configuration Layer
    subgraph "Configuration Layer"
        c1["Inference Config"]:::config
        c2["Stage1 Config"]:::config
        c3["Stage2 Config"]:::config
    end

    %% Data Processing & Preprocessing
    subgraph "Data Processing/Preprocessing"
        subgraph "Datasets"
            dp1["data_utils"]:::dataprocess
            dp2["image_dataset"]:::dataprocess
            dp3["video_dataset"]:::dataprocess
        end
        subgraph "Scripts Data Processors"
            sp1["generate_dwpose"]:::dataprocess
            sp2["generate_smpls"]:::dataprocess
            sp3["smpl_transfer"]:::dataprocess
        end
    end

    %% Model Components
    subgraph "Model Components"
        m1["attention"]:::model
        m2["champ_model"]:::model
        m3["guidance_encoder"]:::model
        m4["motion_module"]:::model
        m5["transformer_2d"]:::model
        m6["transformer_3d"]:::model
        m7["unet_2d_blocks"]:::model
        m8["unet_2d_condition"]:::model
        m9["unet_3d"]:::model
        m10["unet_3d_blocks"]:::model
    end

    %% Pipeline Components
    subgraph "Pipeline Components"
        p1["context"]:::pipeline
        p2["pipeline_aggregation"]:::pipeline
        p3["pipeline_guidance2image"]:::pipeline
        p4["pipeline_guidance2video"]:::pipeline
    end

    %% Training & Inference Entry Points
    subgraph "Training & Inference Entry Points"
        t1["train_s1.py"]:::training
        t2["train_s2.py"]:::training
        i1["inference.py"]:::inference
    end

    %% Utility & Support
    subgraph "Utility & Support"
        u1["fs"]:::utility
        u2["tb_tracker"]:::utility
        u3["util"]:::utility
        u4["video_utils"]:::utility
    end

    %% External Resources
    subgraph "Pretrained Models"
        ext1["download (pretrained)"]:::external
    end

    %% Results
    r1("Results Directory"):::results

    %% Data flow connections
    A --- dp1
    A --- dp2
    A --- dp3
    A --- sp1
    A --- sp2
    A --- sp3

    %% Data Processors to Models
    dp1 --> m1
    sp1 --> m1
    dp3 --> m2
    sp2 --> m3
    sp3 --> m4

    %% Models to Pipeline Components
    m2 --> p1
    m3 --> p3
    m4 --> p2
    m1 --> p4

    %% Configurations to Entry Points
    c2 --> t1
    c3 --> t2
    c1 --> i1

    %% Pipeline Components and Models feed Entry Points
    p3 --> i1
    p2 --> t2

    %% Training flow chain
    t1 --> t2

    %% Entry Points use Utilities
    t1 --- u1
    t1 --- u2
    i1 --- u3
    i1 --- u4

    %% External Resources support Models and Inference
    ext1 --> m2
    ext1 --> i1

    %% Final Output
    i1 --> r1
    p4 --> r1

    %% Click Events for Configuration Files
    click c1 "https://github.com/fudan-generative-vision/champ/blob/master/configs/inference/inference.yaml"
    click c2 "https://github.com/fudan-generative-vision/champ/blob/master/configs/train/stage1.yaml"
    click c3 "https://github.com/fudan-generative-vision/champ/blob/master/configs/train/stage2.yaml"

    %% Click Events for Datasets
    click dp1 "https://github.com/fudan-generative-vision/champ/blob/master/datasets/data_utils.py"
    click dp2 "https://github.com/fudan-generative-vision/champ/blob/master/datasets/image_dataset.py"
    click dp3 "https://github.com/fudan-generative-vision/champ/blob/master/datasets/video_dataset.py"

    %% Click Events for Scripts Data Processors
    click sp1 "https://github.com/fudan-generative-vision/champ/blob/master/scripts/data_processors/dwpose/generate_dwpose.py"
    click sp2 "https://github.com/fudan-generative-vision/champ/blob/master/scripts/data_processors/smpl/generate_smpls.py"
    click sp3 "https://github.com/fudan-generative-vision/champ/blob/master/scripts/data_processors/smpl/smpl_transfer.py"

    %% Click Events for Model Components
    click m1 "https://github.com/fudan-generative-vision/champ/blob/master/models/attention.py"
    click m2 "https://github.com/fudan-generative-vision/champ/blob/master/models/champ_model.py"
    click m3 "https://github.com/fudan-generative-vision/champ/blob/master/models/guidance_encoder.py"
    click m4 "https://github.com/fudan-generative-vision/champ/blob/master/models/motion_module.py"
    click m5 "https://github.com/fudan-generative-vision/champ/blob/master/models/transformer_2d.py"
    click m6 "https://github.com/fudan-generative-vision/champ/blob/master/models/transformer_3d.py"
    click m7 "https://github.com/fudan-generative-vision/champ/blob/master/models/unet_2d_blocks.py"
    click m8 "https://github.com/fudan-generative-vision/champ/blob/master/models/unet_2d_condition.py"
    click m9 "https://github.com/fudan-generative-vision/champ/blob/master/models/unet_3d.py"
    click m10 "https://github.com/fudan-generative-vision/champ/blob/master/models/unet_3d_blocks.py"

    %% Click Events for Pipeline Components
    click p1 "https://github.com/fudan-generative-vision/champ/blob/master/pipelines/context.py"
    click p2 "https://github.com/fudan-generative-vision/champ/blob/master/pipelines/pipeline_aggregation.py"
    click p3 "https://github.com/fudan-generative-vision/champ/blob/master/pipelines/pipeline_guidance2image.py"
    click p4 "https://github.com/fudan-generative-vision/champ/blob/master/pipelines/pipeline_guidance2video.py"

    %% Click Events for Training & Inference Entry Points
    click t1 "https://github.com/fudan-generative-vision/champ/blob/master/train_s1.py"
    click t2 "https://github.com/fudan-generative-vision/champ/blob/master/train_s2.py"
    click i1 "https://github.com/fudan-generative-vision/champ/blob/master/inference.py"

    %% Click Events for Utility & Support
    click u1 "https://github.com/fudan-generative-vision/champ/blob/master/utils/fs.py"
    click u2 "https://github.com/fudan-generative-vision/champ/blob/master/utils/tb_tracker.py"
    click u3 "https://github.com/fudan-generative-vision/champ/blob/master/utils/util.py"
    click u4 "https://github.com/fudan-generative-vision/champ/blob/master/utils/video_utils.py"

    %% Click Event for External Resources
    click ext1 "https://github.com/fudan-generative-vision/champ/blob/master/scripts/pretrained_models/download.py"

    %% Styles
    classDef external fill:#FFD700,stroke:#DAA520,stroke-width:2px;
    classDef dataprocess fill:#AED6F1,stroke:#5DADE2,stroke-width:2px;
    classDef model fill:#D5F5E3,stroke:#58D68D,stroke-width:2px;
    classDef pipeline fill:#FAD7A0,stroke:#F39C12,stroke-width:2px;
    classDef training fill:#A3E4D7,stroke:#117864,stroke-width:2px;
    classDef inference fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef config fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef utility fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;
    classDef results fill:#C39BD3,stroke:#6C3483,stroke-width:2px;
```
