How tencent-hunyuan/hunyuancustom works
Project Type and Purpose
• HunyuanCustom is a full-stack, multimodal AI inference system for customized video generation. It combines:
• A backend inference library (hymm_sp) implementing diffusion pipelines, data preprocessing, model modules (UNet, VAE, text encoders, audio adapters, schedulers, etc.)
• A frontend/web layer (hymm_gradio) exposing CLI scripts and a Gradio/Flask server for user interaction
• A model weight repository (models/) and sample assets for input/output
High-Level File Structure Analysis
• Top-level directories:
• assets/ – sample media inputs, masks, metadata, logos
• hymm_sp/ – core inference package:
– data_kits/ (pose detection, video dataset)
– diffusion/ (pipelines + schedulers)
– modules/ (model building blocks: attention, MLP, VAEs, token refiner)
– text_encoder/, vae/ subpackages
– inference entrypoints (inference.py, sample_batch.py, sample_gpu_poor.py)
• hymm_gradio/ – UI layer (Flask wrappers, Gradio interfaces, end-to-end tools)
• models/ – documentation and downloaded checkpoints
• scripts/ – shell wrappers to launch CLI and Gradio server
Key Insights from README
• Multi-modal conditioning: image/text/audio/video in → customized video out
• Built on top of HunyuanVideo core
• Supports single-GPU/ multi-GPU, Docker, CPU offload, low-VRAM modes
• Various “run_…” scripts wire together user input, model loading, inference, result storage
Main Components
a. User Interfaces
• CLI Runner (shell scripts → sample_gpu_poor.py, sample_batch.py)
• Web Server
– Flask + Gradio endpoints (in hymm_gradio/)
b. Inference Service
• Inference orchestrator (hymm_sp/inference.py)
• Sampling scripts (sample_batch.py, sample_gpu_poor.py)
c. Data Preprocessing
• Pose detection (hymm_sp/data_kits/dwpose)
• Video/Audio loaders and formatters (hymm_sp/data_kits/video_dataset, audio adapters)
d. Model Pipeline
• Multimodal fusion modules (LLaVA-based text-image fusion)
• Identity enhancement module
• AudioNet (audio cross-attention)
• Video injection network (patchify feature align)
• Diffusion pipeline (pipeline_hunyuan_video_custom)
• Core ML modules (UNet_3D, VAE, text encoder, schedulers)
e. Model Store
• Checkpoints (models/)
• Configuration (hymm_sp/config.py)
f. Compute Resources
• GPU cluster (multi-GPU via torchrun, single GPU, CPU offload)
• Docker container (optional runtime)
g. Output & Storage
• Video writer (results directory)
• Gradio UI display / CLI logging
Component Interactions & Data Flow
1) User → CLI or HTTP Request
2) UI Layer (scripts/run_… or hymm_gradio) → Inference Service API
3) Inference Service
– Loads checkpoint from Model Store
– Invokes Data Preprocessing (pose, audio, video loaders)
– Feeds conditioned inputs to the Model Pipeline
– Runs diffusion sampling across GPU(s)
4) Pipeline components communicate via latent tensors and attention maps
5) Generated latent video frames → VAE decoder → final video
6) Output saved to disk and returned to user interface
Architectural Patterns & Principles
• Modular pipeline: separate preprocessing, conditioning injection, diffusion sampling, decoding
• Layered separation: UI (hymm_gradio/scripts) → orchestration (inference entrypoints) → core ML library (hymm_sp/modules + pipelines) → compute
• Plugin-style fusion: different “condition injectors” for text/image/audio/video
• Scalability: torchrun for multi-GPU; CPU offload/fp8 modes for low VRAM