# fudan-generative-vision/hallo2 architecture

> Architecture diagram of the GitHub repository fudan-generative-vision/hallo2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fudan-generative-vision/hallo2
- Repository: https://github.com/fudan-generative-vision/hallo2

## Overview

##### Main Components
- Input Data and Preprocessing:
- Inputs include portrait images and driving audio (WAV files).
- A “Data Preprocessing” module (scripts/data_preprocess.py, hallo/datasets) handles conversion of videos to frames, extraction of audio, generation of masks, face cropping and face embedding creation.
- Core Model Architectures:
- The heart of the project lies in the deep learning modules:
▪ The “basicsr” library contains various network architectures (arcface_arch, codeformer_arch, rrdbnet_arch, vgg_arch, vqgan_arch) plus loss functions, metrics, and model wrappers.
▪ The “hallo” directory hosts specific models (audio_proj, image_proj, face_locator, transformer_2d/3d, attention models, unet-based modules) used for the animation process.
▪ “facelib” provides external face detection and parsing utilities.
- Inference Pipelines:
- Two inference entry points: “inference_long.py” for long-duration audio-driven animation and “video_sr.py” for high-resolution video animation.
- They load pretrained models (downloaded via HuggingFace or organized in the “pretrained_models” directory) and perform the final animation generation.
- Training and Distributed Setup:
- Training scripts (train_stage1.py, train_stage2_long.py, basicsr/train.py) are used for different training phases.
- The project employs distributed training (via Hugging Face Accelerate and torch.distributed.launch) with configuration files (e.g., accelerate_config.yaml and YAML configs in the “configs” folder).
- The data meta extraction and preparation steps (extract_meta_info_stage1.py, extract_meta_info_stage2.py) are also key parts.
- Configuration and Utilities:
- The “configs” directory holds configuration files which parametrize both inference and training.
- A variety of utility modules across “basicsr/utils”, “facelib/utils”, and “hallo/utils” provide logging, file handling, image processing, and other support functions.

##### Representing Relationships and Data Flow
- A pre-processing block:
- A “Data Preprocessor” that extracts frames, audio, face crops, and generates masks.
- Split into two parallel branches:
- One branch flows into “Long-duration Animation Inference”, which uses the core “hallo” models along with audio encoders (e.g., wav2vec from pretrained models) to produce animated portrait sequences.
- The other branch leads to “High-Resolution Animation” where the “basicsr” (and optionally CodeFormer-based) modules, along with “facelib” detection and upsampling (e.g., realesrgan), generate enhanced high-resolution outputs.
- For the Training Pipeline:
- That the training pipeline follows two stages (Stage 1 and Stage 2) as defined in the YAML configs.
- That distributed training is orchestrated by the Accelerate tool (with parameters in accelerate_config.yaml) and by torch.distributed.launch.
- External and Supporting Components:
- An “External Pretrained Models” component that supplies checkpoints for various submodules (face analysis, audio separator, etc.).
- Configuration management (the “configs” folder) and dependencies (e.g., installed Python packages, CUDA libraries, FFmpeg).

##### Architectural Patterns and Design Principles
- Modular Design: The project is split into independent modules (data, models, operations, training scripts) to support both inference and training. Emphasize the clear separation of core processing tasks such as face analysis, audio processing, and animation generation.
- Distributed Training: The design leverages distributed computing frameworks (Accelerate and torch.distributed) – show these as a parallel computing block interacting with training modules.
- Reuse and Extensibility: The use of libraries such as basicsr and facelib and configuration-driven design (YAML files) indicates that the system is built to be extensible.

##### Technologies and Frameworks
- Python and Deep Learning (torch, torchvision).
- CUDA and FFmpeg for GPU-based operations and media processing.
- Hugging Face Accelerate and Torch Distributed Launch for training scalability.
- Several custom architectures and loss functions implemented under basicsr as well as external modules from CodeFormer, AnimeDiff, and others.

##### Diagramming Details for the Principal Software Engineer
– “Data Preprocessor”, “Audio Encoder (wav2vec)”, “Face Detection & Alignment (facelib)”, “Image Restoration (basicsr)”, “Animation Generator (hallo)”, “Training Coordinator (Accelerate)”, etc.
– Show how raw inputs are processed into intermediate representations (e.g., audio embeddings and face features).
– Indicate branch-offs into different training inference pipelines.
- Dependencies:
– Which modules depend on specific pretrained weights or external libraries.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Inputs
    subgraph "Inputs"
        SI["Source Image"]:::input
        DA["Driving Audio (WAV)"]:::input
    end

    %% Data Preprocessing
    subgraph "Data Preprocessing"
        DP["Data Preprocessor"]:::preprocess
        DS["Datasets"]:::preprocess
    end

    %% Inference Pipelines
    subgraph "Inference Pipelines"
        LDI["Long-duration Animation Inference"]:::inference
        HRI["High-Resolution Video Super-resolution Inference"]:::inference
    end

    %% Long-duration Inference Modules
    subgraph "Long-duration Modules"
        HM["Hallo Models"]:::model
        AE["Audio Encoder (wav2vec)"]:::model
    end

    %% High-resolution Inference Modules
    subgraph "High-Resolution Modules"
        BS["basicsr Modules"]:::model
        FL["facelib Tools"]:::model
    end

    %% Inference Outputs
    APA["Animated Portrait Sequence"]:::inference
    HRA["Enhanced High-Res Video"]:::inference

    %% External Components
    EPM["External Pretrained Models"]:::external

    %% Training Pipeline
    subgraph "Training Pipeline"
        TDS["Training Dataset"]:::training
        TS1["Train Stage1"]:::training
        TS2["Train Stage2 Long"]:::training
        BT["basicsr Train"]:::training
        EM1["Extract Meta Stage1"]:::training
        EM2["Extract Meta Stage2"]:::training
        TC["Training Coordinator (Accelerate)"]:::training
        AC["Accelerate Config"]:::training
        DT["Distributed Training (torch.distributed.launch)"]:::training
        TM["Trained Models"]:::training
    end

    %% Configuration and Utilities
    CFG["Configurations (configs folder)"]:::util
    subgraph "Utilities & Tools"
        BU["basicsr/utils"]:::util
        FU["facelib/utils"]:::util
        HU["hallo/utils"]:::util
    end

    %% Connections from Inputs to Preprocessing
    SI --> DP
    DA --> DP
    DP --> DS

    %% Branching from Preprocessing to Inference and Training
    DS --> LDI
    DS --> HRI
    DS --> TDS

    %% Long-duration Inference flow
    HM --> LDI
    AE --> LDI
    LDI --> APA

    %% High-resolution Inference flow
    BS --> HRI
    FL --> HRI
    HRI --> HRA

    %% External Models influence inference modules
    EPM --- HM
    EPM --- AE
    EPM --- BS
    EPM --- FL

    %% Configuration influence external models
    CFG --> EPM

    %% Training Pipeline flow
    TDS --> TS1
    TDS --> TS2
    TDS --> BT
    TS1 --> EM1
    TS2 --> EM2
    TS1 --> TC
    TS2 --> TC
    BT --> TM
    TC --> AC
    TC --> DT
    DT --> TM

    %% Configuration influence training
    CFG --> TS1
    CFG --> TS2
    CFG --> BT
    CFG --> TC

    %% Utilities association with model components
    BU --- BS
    FU --- FL
    HU --- HM

    %% Click Events
    click DP "https://github.com/fudan-generative-vision/hallo2/blob/main/scripts/data_preprocess.py"
    click DS "https://github.com/fudan-generative-vision/hallo2/tree/main/hallo/datasets"
    click HM "https://github.com/fudan-generative-vision/hallo2/tree/main/hallo/models"
    click BS "https://github.com/fudan-generative-vision/hallo2/tree/main/basicsr/archs"
    click BT "https://github.com/fudan-generative-vision/hallo2/blob/main/basicsr/train.py"
    click FL "https://github.com/fudan-generative-vision/hallo2/tree/main/facelib/detection"
    click LDI "https://github.com/fudan-generative-vision/hallo2/blob/main/scripts/inference_long.py"
    click HRI "https://github.com/fudan-generative-vision/hallo2/blob/main/scripts/video_sr.py"
    click TS1 "https://github.com/fudan-generative-vision/hallo2/blob/main/scripts/train_stage1.py"
    click TS2 "https://github.com/fudan-generative-vision/hallo2/blob/main/scripts/train_stage2_long.py"
    click EM1 "https://github.com/fudan-generative-vision/hallo2/blob/main/scripts/extract_meta_info_stage1.py"
    click EM2 "https://github.com/fudan-generative-vision/hallo2/blob/main/scripts/extract_meta_info_stage2.py"
    click AC "https://github.com/fudan-generative-vision/hallo2/blob/main/accelerate_config.yaml"
    click BU "https://github.com/fudan-generative-vision/hallo2/tree/main/basicsr/utils"
    click FU "https://github.com/fudan-generative-vision/hallo2/tree/main/facelib/utils"
    click HU "https://github.com/fudan-generative-vision/hallo2/tree/main/hallo/utils"
    click CFG "https://github.com/fudan-generative-vision/hallo2/tree/main/configs"

%% Style classes
classDef input fill:#FFCCCC,stroke:#CC0000,stroke-width:2px,color:#000;
classDef preprocess fill:#FFF2CC,stroke:#CC9900,stroke-width:2px,color:#000;
classDef inference fill:#D5F5E3,stroke:#229954,stroke-width:2px,color:#000;
classDef training fill:#D6EAF8,stroke:#2874A6,stroke-width:2px,color:#000;
classDef model fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px,color:#000;
classDef util fill:#FAD7A0,stroke:#E67E22,stroke-width:2px,color:#000;
classDef external fill:#D6DBDF,stroke:#5D6D7E,stroke-width:2px,color:#000;
```
