# tencent/hunyuanvideo architecture

> Architecture diagram of the GitHub repository tencent/hunyuanvideo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent/hunyuanvideo
- Repository: https://github.com/tencent/hunyuanvideo

## Overview

##### Project Overview and Purpose
- The HunyuanVideo project is an open‐source framework for high-quality, text‐to‐video generation. It uses a diffusion-based generative architecture on a “spatial-temporally compressed” latent space (via a 3D VAE).
- The README explains its key components (unified generative design for video/image, MLLM text encoder, dual–stream/single–stream diffusion backbone, prompt rewrite, and 3D VAE) and how they work together to generate videos from text prompts.
- The project includes inference scripts (for single GPU and multi-GPU via xDiT), a Gradio-based web demo, and command-line runners.

##### Analyzing the File Structure and Its Implications
- Top level: files such as README, LICENSE, and assets (which include visual demo materials) confirm that the repository is self-contained.
- The core code is under the “hyvideo” directory. Inside:
– config.py and constants.py handle configuration.
– The “diffusion” subdirectory (with pipelines and schedulers) holds the central diffusion modeling logic.
– The “vae” folder contains the 3D VAE code that compresses and decodes video data.
– The “text_encoder” folder indicates the use of a pre-trained MLLM (decoder-only) text encoder.
– The “modules” directory contains additional neural network layers (attention, MLPs, token refining, etc.) to support the unified generative pipeline and possibly extra refinements like bidirectional token refiner.
– “inference.py” suggests a script tying together the components to perform video generation.
– “prompt_rewrite.py” handles rewriting prompts via a specialized model.
- Other files and directories (tests, scripts, gradio_server.py, sample_video.py) show support for testing, execution, and web demo.

##### Main Components
- Input Interface:
- User provided text prompt.
- Optional command-line or Gradio web interface.
- Prompt Rewrite Module (hyvideo/prompt_rewrite.py):
- Processes the raw text prompt to adapt it to model-preferred style (with both Normal and Master modes).
- Text Encoder (hyvideo/text_encoder):
- A pre-trained Multimodal Large Language Model (MLLM) used to convert the (rewritten) prompt into features.
- Diffusion Pipeline (hyvideo/diffusion):
- Contains the core generative model logic.
- Includes “pipeline_hunyuan_video.py” that orchestrates the diffusion process.
- Uses scheduler modules (e.g., scheduling_flow_match_discrete.py) that handle the sampling trajectory.
- Implements both a dual-stream (separate handling of video and text tokens) and single-stream (fused) design.
- 3D VAE (hyvideo/vae):
- Responsible for compressing video frames into a lower-dimensional latent space and then decoding the generated latent back into video frames.
- Supporting Modules:
- Layers and utilities under “hyvideo/modules” (activation, attention, embedding, normalization, etc.) that support the diffusion backbone.
- Utility functions (hyvideo/utils) for data handling and file management.
- Inference Orchestration:
- The “inference.py” file (and scripts like sample_video.py) ties together the modules and triggers the generation process.
- Additional wrappers like “gradio_server.py” provide an interactive front-end to the model.
- Resource Management and Deployment:
- Configuration of multi-GPU inference via xDiT (indicated in the README and scripts) should be represented.
- Pre-trained checkpoints (ckpts directory) as well as additional utility scripts (e.g., for environment collection in utils/collect_env.py).

##### Relationships and Data Flow
- The prompt first goes into the Prompt Rewrite Module, which “cleans” and “adapts” the prompt.
- The rewritten prompt is then routed to the Text Encoder, converting it into feature embeddings.
- In parallel, a noise vector (sampled from a Gaussian distribution) is generated.
- Both the text embedding and the noise vector serve as inputs into the Diffusion Pipeline:
- The model first processes video and text tokens separately in dual‐stream transformer layers.
- Then a single-stream fusion occurs to enable multimodal integration.
- Scheduler modules manage the reverse diffusion steps and evolution of latent variables.
- The generated latent representation is passed to the 3D VAE’s decoder, which reconstructs the video frame sequence.
- The final video is either saved (as commanded via sample_video.py) or served via a web interface (gradio_server.py).

##### Architectural Patterns and Technical Stack
- Modularity: Each major block (prompt rewrite, text encoder, diffusion pipeline, VAE) is encapsulated as an independent module in the code base.
- Pipeline-style Data Flow: Sequential transformation from text prompt to latent generation to video decoding.
- Scalability: Support for multi-GPU parallelism using xDiT and CPU-offload approaches.
- Technology Stack: PyTorch implementation; integration with Diffusers (HuggingFace) for diffusion pipeline interfaces; dependence on CUDA for GPU acceleration.

##### Final Assembly Advice for the Engineer
- Then, expand each block to include sub-components (e.g., break the diffusion block into dual-stream processing, single-stream fusion, and scheduler control).
- Annotations about multi-GPU deployment via xDiT and CPU offloading to save memory.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        UI["User Interface (CLI/Gradio)"]:::interface
        TP["Text Prompt"]:::data
        PR["Prompt Rewrite Module"]:::module
        TE["Text Encoder (MLLM)"]:::module
        UI -->|"submit"| TP
        TP -->|"process"| PR
        PR -->|"rewrite"| TE
    end

    %% Generative Model Layer
    subgraph "Generative Model Layer"
        NG["Noise Generator"]:::data
        DP["Diffusion Pipeline"]:::module
        DT["Dual-Stream Processing"]:::module
        SS["Single-Stream Fusion"]:::module
        SC["Scheduler Control"]:::module
        NG -->|"samples noise"| DP
        TE -->|"embeddings"| DP
        DP --> DT
        DT --> SS
        SS --> SC
    end

    %% Latent Representation and Decoding Layer
    LR["Latent Representation"]:::data
    SC -->|"generates latent"| LR

    subgraph "Decoding Layer"
        VAE["3D VAE Decoder"]:::decoder
        OUT["Output Video Generator"]:::output
        VAE -->|"decodes"| OUT
    end
    LR --> VAE

    %% Inference Orchestration
    INF["Inference Orchestration"]:::orchestrator
    INF --- PR
    INF --- TE
    INF --- DP
    INF --- VAE

    %% Supporting Components
    SUP["Neural Network Modules"]:::support
    CC["Configuration & Constants"]:::support
    UTIL["Utility Functions"]:::support
    CKPT["Pre-trained Checkpoints"]:::support

    SUP ---|"supports"| DP
    CC ---|"configures"| INF
    UTIL ---|"assists"| INF
    CKPT ---|"provides models"| DP

    %% Click Events
    click PR "https://github.com/tencent/hunyuanvideo/blob/main/hyvideo/prompt_rewrite.py"
    click TE "https://github.com/tencent/hunyuanvideo/tree/main/hyvideo/text_encoder/"
    click DP "https://github.com/tencent/hunyuanvideo/blob/main/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py"
    click SC "https://github.com/tencent/hunyuanvideo/blob/main/hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py"
    click VAE "https://github.com/tencent/hunyuanvideo/blob/main/hyvideo/vae/vae.py"
    click SUP "https://github.com/tencent/hunyuanvideo/tree/main/hyvideo/modules/"
    click INF "https://github.com/tencent/hunyuanvideo/blob/main/hyvideo/inference.py"
    click CC "https://github.com/tencent/hunyuanvideo/blob/main/hyvideo/config.py"
    click UTIL "https://github.com/tencent/hunyuanvideo/tree/main/hyvideo/utils/"
    click CKPT "https://github.com/tencent/hunyuanvideo/tree/main/ckpts/"

    %% Styles
    classDef interface fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef data fill:#d6eaf8,stroke:#2e86c1,stroke-width:2px;
    classDef module fill:#fadbd8,stroke:#c0392b,stroke-width:2px;
    classDef decoder fill:#d5f5e3,stroke:#28b463,stroke-width:2px;
    classDef output fill:#ebdef0,stroke:#8e44ad,stroke-width:2px;
    classDef orchestrator fill:#fcf3cf,stroke:#f1c40f,stroke-width:2px;
    classDef support fill:#d7bde2,stroke:#6c3483,stroke-width:2px;
```
