How tencent/hunyuancustom works
Project Type and Purpose
• HunyuanCustom is a full-stack, research-grade video generation framework.
• It provides both a programmatic inference API (Python scripts / library) and a web interface (Gradio/Flask), driven by a diffusion-based, multimodal model that accepts text, image, audio, and video conditions to produce customized videos.
File Structure Analysis
• Top-level:
• assets/ → static resources (images, videos, meta files) used as examples or UI materials
• hymm_sp/ → core Python library implementing model components and inference pipelines
• hymm_gradio/ → lightweight web service wrappers (Flask & Gradio endpoints)
• models/ → placeholder for downloaded pre-trained checkpoints
• scripts/ → shell scripts to launch inference or server
• requirements.txt, Docker instructions, and conda setup files for environment configuration
• hymm_sp subpackages reveal architectural layers:
• config, constants → global settings
• data_kits → data loading and preprocessing (VideoDataset)
• diffusion/pipelines → custom diffusion pipeline
• diffusion/schedulers → scheduler implementation for the denoising process
• modules → neural network building blocks (attention, activation, normalization, MLPs, FP8 optimizations, text-image fusion, AudioNet, patchify-based video conditioning)
• vae → encoder/decoder for latent video representation
• text_encoder → multimodal text/image embedding fusion (LLaVA-based)
• inference.py, sample_*.py → entry points for various inference modes (single-GPU, multi-GPU, low-VRAM)
README Insights
• Explicit mention of four conditioning modalities: text, image, audio, video.
• Two interfaces: CLI scripts (sample_batch.py, sample_gpu_poor.py) and Gradio server (gradio_ref2v.py, gradio_id.py).
Detailed Capturing of Architectural Elements
• Don’t omit the data_kits layer: show how raw assets can feed directly into preprocessing.
• Two distinct inference entry points (sample_gpu_poor.py vs. sample_batch.py) to illustrate resource-adaptive modes.