# tencent-hunyuan/hunyuancustom architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/hunyuancustom, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/hunyuancustom
- Repository: https://github.com/tencent-hunyuan/hunyuancustom

## Overview

##### Project Type and Purpose
- HunyuanCustom is a full-stack, multimodal AI inference system for customized video generation. It combines:
- A backend inference library (hymm_sp) implementing diffusion pipelines, data preprocessing, model modules (UNet, VAE, text encoders, audio adapters, schedulers, etc.)
- A frontend/web layer (hymm_gradio) exposing CLI scripts and a Gradio/Flask server for user interaction
- A model weight repository (models/) and sample assets for input/output

##### High-Level File Structure Analysis
- Top-level directories:
- assets/ – sample media inputs, masks, metadata, logos
- hymm_sp/ – core inference package:
– data_kits/ (pose detection, video dataset)
– diffusion/ (pipelines + schedulers)
– modules/ (model building blocks: attention, MLP, VAEs, token refiner)
– text_encoder/, vae/ subpackages
– inference entrypoints (inference.py, sample_batch.py, sample_gpu_poor.py)
- hymm_gradio/ – UI layer (Flask wrappers, Gradio interfaces, end-to-end tools)
- models/ – documentation and downloaded checkpoints
- scripts/ – shell wrappers to launch CLI and Gradio server

##### Key Insights from README
- Multi-modal conditioning: image/text/audio/video in → customized video out
- Built on top of HunyuanVideo core
- Supports single-GPU/ multi-GPU, Docker, CPU offload, low-VRAM modes
- Various “run_…” scripts wire together user input, model loading, inference, result storage

##### Main Components
a. User Interfaces
- CLI Runner (shell scripts → sample_gpu_poor.py, sample_batch.py)
- Web Server
– Flask + Gradio endpoints (in hymm_gradio/)
b. Inference Service
- Inference orchestrator (hymm_sp/inference.py)
- Sampling scripts (sample_batch.py, sample_gpu_poor.py)
c. Data Preprocessing
- Pose detection (hymm_sp/data_kits/dwpose)
- Video/Audio loaders and formatters (hymm_sp/data_kits/video_dataset, audio adapters)
d. Model Pipeline
- Multimodal fusion modules (LLaVA-based text-image fusion)
- Identity enhancement module
- AudioNet (audio cross-attention)
- Video injection network (patchify feature align)
- Diffusion pipeline (pipeline_hunyuan_video_custom)
- Core ML modules (UNet_3D, VAE, text encoder, schedulers)
e. Model Store
- Checkpoints (models/)
- Configuration (hymm_sp/config.py)
f. Compute Resources
- GPU cluster (multi-GPU via torchrun, single GPU, CPU offload)
- Docker container (optional runtime)
g. Output & Storage
- Video writer (results directory)
- Gradio UI display / CLI logging

##### Component Interactions & Data Flow
1) User → CLI or HTTP Request
2) UI Layer (scripts/run_… or hymm_gradio) → Inference Service API
3) Inference Service
– Loads checkpoint from Model Store
– Invokes Data Preprocessing (pose, audio, video loaders)
– Feeds conditioned inputs to the Model Pipeline
– Runs diffusion sampling across GPU(s)
4) Pipeline components communicate via latent tensors and attention maps
5) Generated latent video frames → VAE decoder → final video
6) Output saved to disk and returned to user interface

##### Architectural Patterns & Principles
- Modular pipeline: separate preprocessing, conditioning injection, diffusion sampling, decoding
- Layered separation: UI (hymm_gradio/scripts) → orchestration (inference entrypoints) → core ML library (hymm_sp/modules + pipelines) → compute
- Plugin-style fusion: different “condition injectors” for text/image/audio/video
- Scalability: torchrun for multi-GPU; CPU offload/fp8 modes for low VRAM

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User
    User["User"] 

    %% UI Layer
    subgraph "UI Layer" 
        CLI["CLI Runner"]:::ui 
        sample_batch["sample_batch.py"]:::ui 
        sample_gpu_poor["sample_gpu_poor.py"]:::ui 
        sample_inference["sample_inference.py"]:::ui 
        Web["Web Server"]:::ui 
        flask_hycustom["flask_hycustom.py"]:::ui 
        flask_ref2v["flask_ref2v.py"]:::ui 
        gradio_audio["gradio_audio.py"]:::ui 
        gradio_editing["gradio_editing.py"]:::ui 
        gradio_ref2v["gradio_ref2v.py"]:::ui 
        tool_end2end["tool_for_end2end.py"]:::ui 
    end

    %% Orchestrator Layer
    subgraph "Inference Orchestrator" 
        inference["inference.py"]:::ml 
        config["config.py"]:::ml 
        constants["constants.py"]:::ml 
    end

    %% Preprocessing Layer
    subgraph "Data Preprocessing" 
        data_tools["data_tools.py"]:::preprocess 
        dwpose_detector["dwpose_detector.py"]:::preprocess 
        video_dataset["video_dataset.py"]:::preprocess 
    end

    %% Model Pipeline
    subgraph "Model Pipeline" 
        pipeline["pipeline_hunyuan_video_custom.py"]:::ml 
        scheduler["scheduling_flow_match_discrete.py"]:::ml 
        audio_adapter["audio_adapter.py"]:::ml 
        token_refiner["token_refiner.py"]:::ml 
        fp8["fp8_optimization.py"]:::ml 
        parallel_states["parallel_states.py"]:::ml 
        subgraph "Core Modules" 
            attn["attn_layers"]:::ml 
            mlp["mlp_layers"]:::ml 
            modulate["modulate_layers"]:::ml 
            posemb["posemb_layers"]:::ml 
        end
        subgraph "Text & VAE" 
            text_enc["text_encoder"]:::ml 
            vae_mod["vae"]:::ml 
        end
    end

    %% Decoder
    VAEDecoder["VAE Decoder"]:::ml 

    %% Storage
    subgraph "Storage" 
        models["Model Store"]:::storage 
        assets["Sample Assets"]:::storage 
        results["Output Results"]:::storage 
    end

    %% Infrastructure
    subgraph "Compute Resources" 
        GPU["GPU Cluster"]:::infra 
        Docker["Docker Container"]:::infra 
    end

    %% Connections
    User -->|CLI input| CLI
    User -->|HTTP request| Web

    CLI --> sample_batch
    CLI --> sample_gpu_poor
    CLI --> sample_inference

    Web --> flask_hycustom
    Web --> flask_ref2v
    Web --> gradio_audio
    Web --> gradio_editing
    Web --> gradio_ref2v
    Web --> tool_end2end

    sample_batch --> inference
    sample_gpu_poor --> inference
    sample_inference --> inference
    flask_hycustom --> inference
    flask_ref2v --> inference
    gradio_audio --> inference
    gradio_editing --> inference
    gradio_ref2v --> inference
    tool_end2end --> inference

    inference -->|load config| config
    inference -->|load constants| constants
    models -->|checkpoint load| inference

    inference -->|invoke| data_tools
    inference -->|invoke| dwpose_detector
    inference -->|invoke| video_dataset

    data_tools --> pipeline
    dwpose_detector --> pipeline
    video_dataset --> pipeline

    pipeline --> scheduler
    pipeline --> audio_adapter
    pipeline --> token_refiner
    pipeline --> fp8
    pipeline --> parallel_states
    attn --> pipeline
    mlp --> pipeline
    modulate --> pipeline
    posemb --> pipeline
    text_enc --> pipeline
    vae_mod --> pipeline

    pipeline -->|latent frames| VAEDecoder
    VAEDecoder -->|video file| results
    results -->|display| Web
    results -->|log| CLI

    GPU -->|compute| pipeline
    Docker -->|runtime| inference

    assets -->|examples| data_tools

    %% Click Events
    click CLI "https://github.com/tencent-hunyuan/hunyuancustom/tree/main/scripts/"
    click sample_batch "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/sample_batch.py"
    click sample_gpu_poor "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/sample_gpu_poor.py"
    click sample_inference "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/sample_inference.py"
    click flask_hycustom "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_gradio/flask_hycustom.py"
    click flask_ref2v "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_gradio/flask_ref2v.py"
    click gradio_audio "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_gradio/gradio_audio.py"
    click gradio_editing "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_gradio/gradio_editing.py"
    click gradio_ref2v "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_gradio/gradio_ref2v.py"
    click tool_end2end "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_gradio/tool_for_end2end.py"
    click inference "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/inference.py"
    click config "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/config.py"
    click constants "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/constants.py"
    click data_tools "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/data_kits/data_tools.py"
    click dwpose_detector "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/data_kits/dwpose/dwpose_detector.py"
    click video_dataset "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/data_kits/video_dataset.py"
    click pipeline "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/diffusion/pipelines/pipeline_hunyuan_video_custom.py"
    click scheduler "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/diffusion/schedulers/scheduling_flow_match_discrete.py"
    click audio_adapter "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/modules/audio_adapter.py"
    click token_refiner "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/modules/token_refiner.py"
    click fp8 "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/modules/fp8_optimization.py"
    click parallel_states "https://github.com/tencent-hunyuan/hunyuancustom/blob/main/hymm_sp/modules/parallel_states.py"
    click models "https://github.com/tencent-hunyuan/hunyuancustom/tree/main/models/"
    click assets "https://github.com/tencent-hunyuan/hunyuancustom/tree/main/assets/"

    %% Styles
    classDef ui fill:#D0E6FF,stroke:#000
    classDef preprocess fill:#D0FFD6,stroke:#000
    classDef ml fill:#FFECB3,stroke:#000
    classDef storage fill:#E0D0FF,stroke:#000
    classDef infra fill:#E0E0E0,stroke:#000
```
