# tencent-hunyuan/hunyuanvideo architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/hunyuanvideo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/hunyuanvideo
- Repository: https://github.com/tencent-hunyuan/hunyuanvideo

## Overview

##### Project Type and Purpose
- HunyuanVideo is a full-stack open-source text-to-video generation framework built in PyTorch.
- Purpose: provide model definitions, training/inference pipelines, 3D VAE compression, prompt rewriting, and multi-GPU/FP8 accelerated sampling (via xDiT/flash-attention) for large video foundation models.

##### File Structure Analysis
- Top-level folders:
- assets, ckpts – static resources, pretrained weights
- hyvideo/ – core library
- scripts/ – shell wrappers for demo runs
- tests/ – unit tests
- utils/ – environment‐collection utilities
- hyvideo/ submodules:
- config.py, constants.py – global settings
- diffusion/ – sampling pipelines & schedulers (pipeline_hunyuan_video, scheduling_flow_match_discrete)
- modules/ – transformer building blocks (attention, MLP, normalization, embeddings, modulations, token refiner)
- text_encoder/ – MLLM wrapper and bidirectional refiner
- vae/ – 3D VAE encoder/decoder definitions
- prompt_rewrite.py – user prompt adaptation logic
- inference.py – high‐level inference orchestration
- utils/ – data I/O, file operations, helper routines
- Entry points:
- sample_video.py – CLI sampling
- gradio_server.py – web demo server

##### README Insights
- Architecture overview: causal 3D VAE compresses spatial–temporal latents → MLLM text encoder (+ token refiner) generates conditioning embeddings → diffusion transformer (dual‐stream → single‐stream) performs denoising → 3D VAE decoder reconstructs video.
- Key features: unified transformer backbone, MLLM vs CLIP/T5, 3D VAE compression, prompt rewrite, parallel/FP8 inference.

a. Main Components
1) User Interface
- CLI (sample_video.py), Gradio server (gradio_server.py)
2) Prompt Rewrite Module (hyvideo/prompt_rewrite.py)
3) Text Encoder
- MLLM decoder-only model
- Token Refiner (bidirectional enhancement)
4) 3D VAE
- Encoder: pixel/video → latent
- Decoder: latent → pixel/video
5) Diffusion Transformer
- Dual-stream (separate video & text token blocks)
- Single-stream (fused multimodal blocks)
6) Scheduler / Flow Matching
- scheduling_flow_match_discrete
7) Checkpoints & Weight Management
- ckpts/ directory, FP8 vs full precision
8) Accelerators & Parallel Engines
- flash-attention, xDiT
9) Output
- Generated video file saved to disk or served by Gradio
10) Supporting Utilities
- config.py, constants.py, data_utils, file_utils

b. Interactions (Directional Data Flow)
User Input (prompt, flags) ─► Prompt Rewrite ─► Text Encoder ─► Token Refiner ─►
┌────────────────────────────────────────────────────────────────────┐
│       Diffusion Transformer Pipeline  ↔  Scheduler                 │
│           ↑                                                       │
│    3D VAE Encoder (initial latent compression)                    │
│           ↓                                                       │
│    3D VAE Decoder (final video reconstruction)                    │
└────────────────────────────────────────────────────────────────────┘
─► Output Video (disk/Gradio UI)

c. Architectural Patterns
- Modular Python package: clear separation of model, utilities, I/O
- Transformer-based diffusion with dual→single stream fusion
- Compression-decompression loop via causal 3D VAE
- Pluggable scheduler & precision (FP32/FP8) choices
- Multi-GPU sequence parallelism (xDiT)

d. Key Technologies & Frameworks
- PyTorch 2.4.0 (CUDA 11.8/12.4)
- flash-attention v2
- xDiT unified sequence parallelism
- Gradio for web UI
- HuggingFace Diffusers compatibility

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface Layer
    subgraph "User Interface"
        CLI["CLI: sample_video.py"]:::frontend
        Gradio["Gradio Server: gradio_server.py"]:::frontend
    end

    %% Prompt Rewrite
    PR["Prompt Rewrite Module"]:::core

    %% Text Encoding
    subgraph "Text Encoding"
        TE["MLLM Wrapper"]:::core
        TR["Token Refiner"]:::core
    end

    %% VAE Compression
    subgraph "3D VAE Compression"
        VAE_Enc["3D VAE Encoder"]:::core
        VAE_Dec["3D VAE Decoder"]:::core
    end

    %% Diffusion Pipeline
    subgraph "Diffusion Transformer Pipeline"
        DT["Diffusion Backbone"]:::core
        SCH["Scheduler / Flow Matching"]:::core
    end

    %% Precision & Acceleration
    subgraph "Acceleration & Precision"
        FP8["FP8 & Precision Optimizer"]:::accel
        ACC["xDiT / flash-attn Engine"]:::accel
    end

    %% Configuration & Utilities
    subgraph "Config & Utilities"
        CFG["config.py / constants.py"]:::util
        UT["Supporting Utilities"]:::util
    end

    %% Storage & Output
    CK["Checkpoints: ckpts/"]:::storage
    OUT["Output Video"]:::storage

    %% Data Flow
    CLI -->|inputs| PR
    Gradio -->|inputs| PR
    PR -->|rewritten prompt| TE
    TE -->|embeddings| TR
    TR -->|tokens| VAE_Enc
    VAE_Enc -->|latent| DT
    DT -->|denoised latent| VAE_Dec
    VAE_Enc --> DT
    DT --> SCH
    SCH --> DT
    DT -->|reconstructed frames| VAE_Dec
    VAE_Dec -->|video| OUT

    %% Precision & Accel integration
    DT -->|precision config| FP8
    DT -->|accelerated ops| ACC

    %% Config & Utils usage
    CFG --> PR
    CFG --> TE
    CFG --> VAE_Enc
    CFG --> DT
    UT --> PR
    UT --> TE
    UT --> VAE_Enc
    UT --> DT

    %% Checkpoints flow
    CK --> DT
    CK --> VAE_Enc
    CK --> VAE_Dec

    %% Click Events
    click CLI "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/sample_video.py"
    click Gradio "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/gradio_server.py"
    click PR "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/prompt_rewrite.py"
    click TE "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/text_encoder/__init__.py"
    click TR "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/token_refiner.py"
    click VAE_Enc "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/vae/autoencoder_kl_causal_3d.py"
    click VAE_Enc "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/vae/unet_causal_3d_blocks.py"
    click VAE_Dec "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/vae/vae.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/models.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/attention.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/mlp_layers.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/embed_layers.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/norm_layers.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/modulate_layers.py"
    click DT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/posemb_layers.py"
    click SCH "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py"
    click FP8 "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/modules/fp8_optimization.py"
    click CFG "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/config.py"
    click CFG "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/constants.py"
    click UT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/utils/data_utils.py"
    click UT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/utils/file_utils.py"
    click UT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/utils/helpers.py"
    click UT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py"
    click CK "https://github.com/tencent-hunyuan/hunyuanvideo/tree/main/ckpts/"
    click UT "https://github.com/tencent-hunyuan/hunyuanvideo/blob/main/utils/collect_env.py"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#005B96,color:#000
    classDef core fill:#E0FFE0,stroke:#008000,color:#000
    classDef accel fill:#FFE8C4,stroke:#FF8C00,color:#000
    classDef util fill:#F0F0F0,stroke:#888,color:#000
    classDef storage fill:#FFE0E0,stroke:#C00000,color:#000
```
