# tencent-hunyuan/hunyuanvideo-avatar architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/hunyuanvideo-avatar, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/hunyuanvideo-avatar
- Repository: https://github.com/tencent-hunyuan/hunyuanvideo-avatar

## Overview

##### Project type and purpose
- This is an audio-driven human animation inference engine (HunyuanVideo-Avatar). It provides both CLI scripts and a lightweight web interface (Gradio/Flask) to generate multi-character, emotion-aware talking head videos from avatar images and audio.
- It’s not a full-stack web app in the traditional sense but a specialized ML inference service with optional web front-end.

##### File structure
- Top-level directories:
- hymm_sp: the core Python package implementing model loading, data preprocessing, diffusion pipelines, VAE, text encoder, modules, and inference routines.
- hymm_gradio: two server wrappers (flask_audio.py, gradio_audio.py) plus a tool_for_end2end integration script.
- scripts: shell wrappers to launch Gradio (`run_gradio.sh`), batch inference (`run_sample_batch_sp.sh`), single-GPU and low-VRAM runs.
- weights: pretrained model checkpoints.
- requirements.txt and typical deployment files (.gitignore, Docker/Conda instructions in README).

##### README for additional insights
- Core architecture is a multimodal diffusion transformer (MM-DiT).
- Three modules: Character Image Injection, Audio Emotion Module (AEM), and Face-Aware Audio Adapter (FAA).
- Supports parallel inference with torchrun and single-GPU/low-VRAM modes.
- Offers a Gradio-based UI for interactive demos.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface Layer
    subgraph "User Interface" 
        direction TB
        CLI["CLI Scripts"]:::frontend
        WebUI["Web UI Wrappers"]:::frontend
    end

    %% Orchestrator Layer
    subgraph "Orchestrator / Controller"
        direction TB
        Orchestrator["tool_for_end2end.py"]:::frontend
    end

    %% Core Inference Library Layer
    subgraph "Core Inference Library (hymm_sp)"
        direction TB
        Config["Config & Constants"]:::core
        subgraph "Data Kits (Preprocessing)"
            direction TB
            AudioDataset["audio_dataset.py"]:::core
            AudioPreproc["audio_preprocessor.py"]:::core
            DataTools["data_tools.py"]:::core
            subgraph "Face Alignment"
                direction TB
                Align["align.py"]:::core
                Detface["detface.py"]:::core
            end
        end
        TextEnc["Text Encoder"]:::core
        subgraph "VAE & U-Net Blocks"
            direction TB
            VAECore["vae.py"]:::core
            AutoKL["autoencoder_kl_causal_3d.py"]:::core
            UNet{"unet_causal_3d_blocks.py"}:::core
        end
        subgraph "Diffusion Pipelines & Schedulers"
            direction TB
            Pipeline["pipeline_hunyuan_video_audio.py"]:::core
            Scheduler["scheduling_flow_match_discrete.py"]:::core
            subgraph "Modules & Adapters"
                direction TB
                AudioAdapters["audio_adapters.py"]:::core
                AttnLayers["attn_layers.py"]:::core
                Modulate["modulate_layers.py"]:::core
                TokenRefiner["token_refiner.py"]:::core
            end
        end
        subgraph "Inference Entrypoints"
            direction TB
            InferenceLib["inference.py"]:::core
            SampleBatch["sample_batch.py"]:::core
            SamplePoor["sample_gpu_poor.py"]:::core
            SampleAudio["sample_inference_audio.py"]:::core
        end
    end

    %% Resources and Output
    Weights[(Weights Store)]:::resource
    Compute["Compute Resources\n(GPU/CPU)"]:::resource
    Output["Generated Video"]:::resource

    %% Connections
    CLI -->|invokes| Orchestrator
    WebUI -->|HTTP Request| Orchestrator
    Orchestrator -->|delegates| InferenceLib
    InferenceLib --> Config
    InferenceLib --> AudioDataset
    InferenceLib --> AudioPreproc
    InferenceLib --> DataTools
    InferenceLib --> Align
    InferenceLib --> Detface
    InferenceLib --> TextEnc
    InferenceLib --> VAECore
    InferenceLib --> AutoKL
    InferenceLib --> UNet
    InferenceLib --> Pipeline
    InferenceLib --> Scheduler
    InferenceLib --> AudioAdapters
    InferenceLib --> AttnLayers
    InferenceLib --> Modulate
    InferenceLib --> TokenRefiner
    InferenceLib --> SampleBatch
    InferenceLib --> SamplePoor
    InferenceLib --> SampleAudio

    Pipeline -->|loads| Weights
    Pipeline --> AudioAdapters
    Pipeline --> AttnLayers
    Pipeline --> Modulate
    Pipeline --> TokenRefiner
    Pipeline --> Scheduler
    Pipeline --> UNet
    UNet --> VAECore
    VAECore --> Output
    Compute -->|runs| InferenceLib
    Compute -->|runs| Pipeline

    %% Click Events
    click CLI "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/scripts/run_gradio.sh"
    click CLI "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/scripts/run_sample_batch_sp.sh"
    click CLI "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/scripts/run_single_audio.sh"
    click CLI "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/scripts/run_single_poor.sh"
    click WebUI "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_gradio/gradio_audio.py"
    click WebUI "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_gradio/flask_audio.py"
    click Orchestrator "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_gradio/tool_for_end2end.py"
    click Config "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/config.py"
    click Config "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/constants.py"
    click AudioDataset "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/data_kits/audio_dataset.py"
    click AudioPreproc "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/data_kits/audio_preprocessor.py"
    click DataTools "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/data_kits/data_tools.py"
    click Align "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/data_kits/face_align/align.py"
    click Detface "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/data_kits/face_align/detface.py"
    click TextEnc "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/text_encoder/__init__.py"
    click AutoKL "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/vae/autoencoder_kl_causal_3d.py"
    click UNet "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/vae/unet_causal_3d_blocks.py"
    click VAECore "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/vae/vae.py"
    click Pipeline "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/diffusion/pipelines/pipeline_hunyuan_video_audio.py"
    click Scheduler "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/diffusion/schedulers/scheduling_flow_match_discrete.py"
    click AudioAdapters "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/modules/audio_adapters.py"
    click AttnLayers "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/modules/attn_layers.py"
    click Modulate "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/modules/modulate_layers.py"
    click TokenRefiner "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/modules/token_refiner.py"
    click InferenceLib "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/inference.py"
    click SampleBatch "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/sample_batch.py"
    click SamplePoor "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/sample_gpu_poor.py"
    click SampleAudio "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/blob/main/hymm_sp/sample_inference_audio.py"
    click Weights "https://github.com/tencent-hunyuan/hunyuanvideo-avatar/tree/main/weights/"

    %% Styles
    classDef frontend fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1;
    classDef core fill:#C8E6C9,stroke:#388E3C,color:#1B5E20;
    classDef resource fill:#FFE0B2,stroke:#F57C00,color:#E65100;
    class CLI,WebUI,Orchestrator frontend
    class Config,AudioDataset,AudioPreproc,DataTools,Align,Detface,TextEnc,VAECore,AutoKL,UNet,Pipeline,Scheduler,AudioAdapters,AttnLayers,Modulate,TokenRefiner,InferenceLib,SampleBatch,SamplePoor,SampleAudio core
    class Weights,Compute,Output resource
```
