How tencent-hunyuan/hunyuanvideo works
Project Type and Purpose
• HunyuanVideo is a full-stack open-source text-to-video generation framework built in PyTorch.
• Purpose: provide model definitions, training/inference pipelines, 3D VAE compression, prompt rewriting, and multi-GPU/FP8 accelerated sampling (via xDiT/flash-attention) for large video foundation models.
File Structure Analysis
• Top-level folders:
• assets, ckpts – static resources, pretrained weights
• hyvideo/ – core library
• scripts/ – shell wrappers for demo runs
• tests/ – unit tests
• utils/ – environment‐collection utilities
• hyvideo/ submodules:
• config.py, constants.py – global settings
• diffusion/ – sampling pipelines & schedulers (pipeline_hunyuan_video, scheduling_flow_match_discrete)
• modules/ – transformer building blocks (attention, MLP, normalization, embeddings, modulations, token refiner)
• text_encoder/ – MLLM wrapper and bidirectional refiner
• vae/ – 3D VAE encoder/decoder definitions
• prompt_rewrite.py – user prompt adaptation logic
• inference.py – high‐level inference orchestration
• utils/ – data I/O, file operations, helper routines
• Entry points:
• sample_video.py – CLI sampling
• gradio_server.py – web demo server
README Insights
• Architecture overview: causal 3D VAE compresses spatial–temporal latents → MLLM text encoder (+ token refiner) generates conditioning embeddings → diffusion transformer (dual‐stream → single‐stream) performs denoising → 3D VAE decoder reconstructs video.
• Key features: unified transformer backbone, MLLM vs CLIP/T5, 3D VAE compression, prompt rewrite, parallel/FP8 inference.
a. Main Components
1) User Interface
• CLI (sample_video.py), Gradio server (gradio_server.py)
2) Prompt Rewrite Module (hyvideo/prompt_rewrite.py)
3) Text Encoder
• MLLM decoder-only model
• Token Refiner (bidirectional enhancement)
4) 3D VAE
• Encoder: pixel/video → latent
• Decoder: latent → pixel/video
5) Diffusion Transformer
• Dual-stream (separate video & text token blocks)
• Single-stream (fused multimodal blocks)
6) Scheduler / Flow Matching
• scheduling_flow_match_discrete
7) Checkpoints & Weight Management
• ckpts/ directory, FP8 vs full precision
8) Accelerators & Parallel Engines
• flash-attention, xDiT
9) Output
• Generated video file saved to disk or served by Gradio
10) Supporting Utilities
• config.py, constants.py, data_utils, file_utils
b. Interactions (Directional Data Flow)
User Input (prompt, flags) ─► Prompt Rewrite ─► Text Encoder ─► Token Refiner ─►
┌────────────────────────────────────────────────────────────────────┐
│ Diffusion Transformer Pipeline ↔ Scheduler │
│ ↑ │
│ 3D VAE Encoder (initial latent compression) │
│ ↓ │
│ 3D VAE Decoder (final video reconstruction) │
└────────────────────────────────────────────────────────────────────┘
─► Output Video (disk/Gradio UI)
c. Architectural Patterns
• Modular Python package: clear separation of model, utilities, I/O
• Transformer-based diffusion with dual→single stream fusion
• Compression-decompression loop via causal 3D VAE
• Pluggable scheduler & precision (FP32/FP8) choices
• Multi-GPU sequence parallelism (xDiT)
d. Key Technologies & Frameworks
• PyTorch 2.4.0 (CUDA 11.8/12.4)
• flash-attention v2
• xDiT unified sequence parallelism
• Gradio for web UI
• HuggingFace Diffusers compatibility