How fudan-generative-vision/champ works
Project Type and Purpose
• This project is a research-oriented deep learning framework for human image animation using 3D parametric guidance. It is not a typical full-stack application but rather a pipeline that involves data processing, model building, training, and inference.
• The README, along with the file tree, shows two main workflows: inference (animation generation) and training (with its two stages).
File Structure to Identify Main Components
• Configuration and Workflow:
• The “configs” directory (with separate configs for training [stage1.yaml, stage2.yaml] and inference [inference.yaml]) drives the behavior of both training and inference.
• Data Modules and Preprocessing:
• “datasets” contains dataset handling scripts (video, image datasets, data utilities).
• “scripts/data_processors” includes specialized SMPL and DWPose processing scripts (for blending, generating SMPL data and visualization).
• Model Modules:
• “models” contains various network architectures such as attention, transformers (2D and 3D), UNet blocks, motion modules, etc.
• Pipeline Components:
• “pipelines” provides functions for aggregating contexts and bridging between guidance inputs and final outputs (e.g., pipeline_guidance2image, pipeline_guidance2video).
• Entry Points:
• “inference.py” is the entrypoint for the inference pipeline.
• “train_s1.py” and “train_s2.py” are separate scripts for the two-stage training process.
• Utility and Support:
• “utils” contains file system helpers, tracker for TensorBoard, and video processing utilities.
• “docs”, “assets”, and “images” provide documentation, visual assets, and Docker/containers for deployment support.
• Data Processing & Preprocessing:
• Data sources (training videos, motion samples from external preparation such as SMPL & DWPose).
• Dedicated data processor scripts (e.g., generate_dwpose, generate_smpls, smpl_transfer).
• Configuration Layer:
• Config files for inference and training that guide behavior.
• Model Components:
• Different model blocks (attention net, transformer layers, UNet-based modules, motion module, guidance encoders) from the “models” directory.
• Pipeline Components:
• Components from “pipelines” that combine input data and model outputs to generate final animation.
• Training and Inference Entrypoints:
• “train_s1.py” and “train_s2.py” for training stages.
• “inference.py” for running inference, including handling of pretrained models, guidance data, and result generation.
• External Dependencies:
• Pretrained models (downloaded from HuggingFace and organized under “pretrained_models”).
• External frameworks like Conda, Poetry, Accelerate, and underlying libraries (PyTorch, diffusers, etc.).
b. Relationships and Data Flow:
• Data Flow: Training data (video frames, depth, dwpose, mask, normal, semantic maps) and guidance motion data flow into the preprocessing pipelines (located in scripts and datasets).
• The configuration files define parameters for both training (stage1 and stage2) and inference which then control how data is fed into the pipelines and subsequently into the models.
• After processing, the data enters the model components, feeding into training routines (train_s1.py → train_s2.py) and inference (inference.py).
• Outputs flow from the model and pipelines towards generating animations (stored in a “results” folder) or updating pretrained model checkpoints.
Architectural Patterns
• Modularity: The project is clearly divided into modules (datasets, models, pipelines, utils, scripts) which allows separate understanding and development of each component.
• Two-Stage Training Pipeline: The training process is separated into two stages, with different configs and entrypoints allowing for progressive refinement.
• Data Flow Driven via YAML Configurations: Centralized configuration files provide flexibility and reduce hard-coded dependencies across components.
Final Touches
• Keep the design clean yet comprehensive: major blocks/components should be visible along with interconnections, so that stakeholders and engineers can understand how each module contributes to the whole system.