How fudan-generative-vision/hallo2 works
Main Components
• Input Data and Preprocessing:
• Inputs include portrait images and driving audio (WAV files).
• A “Data Preprocessing” module (scripts/data_preprocess.py, hallo/datasets) handles conversion of videos to frames, extraction of audio, generation of masks, face cropping and face embedding creation.
• Core Model Architectures:
• The heart of the project lies in the deep learning modules:
▪ The “basicsr” library contains various network architectures (arcface_arch, codeformer_arch, rrdbnet_arch, vgg_arch, vqgan_arch) plus loss functions, metrics, and model wrappers.
▪ The “hallo” directory hosts specific models (audio_proj, image_proj, face_locator, transformer_2d/3d, attention models, unet-based modules) used for the animation process.
▪ “facelib” provides external face detection and parsing utilities.
• Inference Pipelines:
• Two inference entry points: “inference_long.py” for long-duration audio-driven animation and “video_sr.py” for high-resolution video animation.
• They load pretrained models (downloaded via HuggingFace or organized in the “pretrained_models” directory) and perform the final animation generation.
• Training and Distributed Setup:
• Training scripts (train_stage1.py, train_stage2_long.py, basicsr/train.py) are used for different training phases.
• The project employs distributed training (via Hugging Face Accelerate and torch.distributed.launch) with configuration files (e.g., accelerate_config.yaml and YAML configs in the “configs” folder).
• The data meta extraction and preparation steps (extract_meta_info_stage1.py, extract_meta_info_stage2.py) are also key parts.
• Configuration and Utilities:
• The “configs” directory holds configuration files which parametrize both inference and training.
• A variety of utility modules across “basicsr/utils”, “facelib/utils”, and “hallo/utils” provide logging, file handling, image processing, and other support functions.
Representing Relationships and Data Flow
• A pre-processing block:
• A “Data Preprocessor” that extracts frames, audio, face crops, and generates masks.
• Split into two parallel branches:
• One branch flows into “Long-duration Animation Inference”, which uses the core “hallo” models along with audio encoders (e.g., wav2vec from pretrained models) to produce animated portrait sequences.
• The other branch leads to “High-Resolution Animation” where the “basicsr” (and optionally CodeFormer-based) modules, along with “facelib” detection and upsampling (e.g., realesrgan), generate enhanced high-resolution outputs.
• For the Training Pipeline:
• That the training pipeline follows two stages (Stage 1 and Stage 2) as defined in the YAML configs.
• That distributed training is orchestrated by the Accelerate tool (with parameters in accelerate_config.yaml) and by torch.distributed.launch.
• External and Supporting Components:
• An “External Pretrained Models” component that supplies checkpoints for various submodules (face analysis, audio separator, etc.).
• Configuration management (the “configs” folder) and dependencies (e.g., installed Python packages, CUDA libraries, FFmpeg).
Architectural Patterns and Design Principles
• Modular Design: The project is split into independent modules (data, models, operations, training scripts) to support both inference and training. Emphasize the clear separation of core processing tasks such as face analysis, audio processing, and animation generation.
• Distributed Training: The design leverages distributed computing frameworks (Accelerate and torch.distributed) – show these as a parallel computing block interacting with training modules.
• Reuse and Extensibility: The use of libraries such as basicsr and facelib and configuration-driven design (YAML files) indicates that the system is built to be extensible.
Technologies and Frameworks
• Python and Deep Learning (torch, torchvision).
• CUDA and FFmpeg for GPU-based operations and media processing.
• Hugging Face Accelerate and Torch Distributed Launch for training scalability.
• Several custom architectures and loss functions implemented under basicsr as well as external modules from CodeFormer, AnimeDiff, and others.
Diagramming Details for the Principal Software Engineer
– “Data Preprocessor”, “Audio Encoder (wav2vec)”, “Face Detection & Alignment (facelib)”, “Image Restoration (basicsr)”, “Animation Generator (hallo)”, “Training Coordinator (Accelerate)”, etc.
– Show how raw inputs are processed into intermediate representations (e.g., audio embeddings and face features).
– Indicate branch-offs into different training inference pipelines.
• Dependencies:
– Which modules depend on specific pretrained weights or external libraries.