How tencent-hunyuan/hunyuanvideo-i2v works
1. Repo purpose and shape
- This is a PyTorch image-to-video generation system built around a pretrained HunyuanVideo base model.
- The repository is split into three clear planes: inference/sampling, LoRA fine-tuning, and data preprocessing / VAE latent extraction.
- It is CLI-first, not a service or web application.
2. Primary entrypoints
- sample_image2video.py is the main inference entrypoint for both standard I2V and LoRA-augmented generation.
- train_image2video_lora.py is the fine-tuning entrypoint for effect-specific LoRA training.
- Shell wrappers in scripts/ orchestrate common runs, including dynamic sampling, stability mode, LoRA training, and multi-GPU execution.
3. Core generation pipeline
- The central runtime path is hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py.
- This pipeline connects:
- the reference image and text prompt,
- the text/image tokenization and embedding stack,
- the video latent generator,
- and the flow-matching sampler / scheduler.
- The output is a decoded video saved to disk, typically as an MP4.
4. Conditioning and multimodal token flow
- The README describes the key I2V design: the reference image is converted into semantic image tokens by a pretrained multimodal encoder.
- Those image tokens are concatenated with video latent tokens so the transformer can attend across both modalities.
- This is the main architectural boundary: conditioning stage first, then joint generation over combined image/video tokens.
5. Model backbone
- hyvideo/modules/ contains the transformer-style building blocks: attention, MLP, embeddings, modulation, normalization, positional embeddings, token refinement, and activation layers.
- hyvideo/modules/models.py is the likely assembly point for the denoising / video backbone.
- hyvideo/modules/fp8_optimization.py suggests optional low-precision acceleration paths for inference efficiency.
6. Latent video representation
- The VAE subsystem in hyvideo/vae/ handles encoding images/video frames into latent space and decoding generated latents back to pixels.
- autoencoder_kl_causal_3d.py and unet_causal_3d_blocks.py indicate a causal 3D VAE design suited to video temporal structure.
- This means generation happens in a compressed latent domain, not directly in pixel space.
7. Sampling, flow matching, and scheduler
- hyvideo/diffusion/flow/ contains the math for the model’s sampling trajectory: transport, paths, integrators, and helpers.
- hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py is the discrete scheduler used during inference.
- In practice, the scheduler drives iterative denoising / flow steps from noise toward a video latent, with options like flow-reverse and flow-shift exposed in the CLI.
8. Text and image preprocessing
- hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py and the hyvideo/text_encoder/ package are the boundary between raw text and model-ready tokens.
- hyvideo/utils/data_utils.py, helpers.py, and file_utils.py support prompt/image loading, resizing, saving, and general orchestration.
- hyvideo/constants.py and config.py centralize model and runtime configuration.
9. LoRA customization path
- LoRA support is integrated into the inference path via hyvideo/utils/lora_utils.py.
- The repository’s LoRA workflow is meant for effect-specific style/motion adaptation, such as “hair growth” or “embrace”.
- At runtime, LoRA weights are loaded and fused into the base model with a user-controlled scale, without changing the base checkpoint.
10. LoRA training subsystem
- train_image2video_lora.py uses the same base model components but optimizes only the LoRA adapters.
- hyvideo/utils/train_utils.py and hyvideo/ds_config.py indicate training support for distributed / memory-efficient setups.
- Training consumes prepared video-caption pairs and produces LoRA checkpoint artifacts, which are later injected during inference.
11. Data preprocessing and VAE extraction
- hyvideo/hyvae_extract/ is a separate preprocessing pipeline for building training data.
- It reads video datasets, runs VAE-related extraction, and emits processed metadata/artifacts used by LoRA training.
- This is an important boundary: raw videos → extracted/processed latent training data → LoRA training.
12. Dataset loading
- hyvideo/dataset/video_loader.py is the repository’s data ingestion layer for video samples.
- It supports the training/preprocessing workflows rather than serving as a generic dataset library.
- The assets/demo/ tree provides example images, videos, and training-data layout for the documented workflows.
13. Multi-GPU inference path
- The repo supports xDiT-based parallel inference through torchrun and sequence parallelism.
- This is an alternate execution mode of the same inference pipeline, not a separate model.
- The architecture changes at runtime by splitting latent computation across GPUs, with ALLOW_RESIZE_FOR_SP=1 enabling size adjustments for divisibility.
14. Operational dependencies and artifacts
- The project depends on PyTorch, CUDA, flash-attn, and optionally xDiT/xfuser for speed and memory efficiency.
- Checkpoints are managed under ckpts/, and the README points to Hugging Face / downloadable weights.
- Generated outputs are written to user-selected result directories; there is no persistent backend service or database layer.
15. How to draw the main architecture
- A useful diagram would show:
- Input image + prompt
- → text/image encoding
- → joint tokenization with video latents
- → diffusion/flow transformer backbone
- → scheduler/integrator loop
- → VAE decoder
- → output video
- Beside that main lane, show a parallel LoRA training lane fed by preprocessed video datasets and producing adapter weights that plug back into the same inference pipeline.
Main entry points: pipeline_hunyuan_video.py I2V pipeline · train_image2video_lora.py Train LoRA · run.py VAE extract · sample_image2video.py Sample I2V
Components 19
CLI & inference
User-facing sampling entrypoints and runtime orchestration.
- Sample I2V · Main inference entrypoint for standard and LoRA-augmented image-to-video generation.
sample_image2video.py - Run scripts · Shell launchers for sampling, stability mode, LoRA runs, and multi-GPU execution.
scripts - Inference glue · Shared inference helpers that assemble model loading, prompt handling, and generation execution.
hyvideo/inference.py - Data utils · Shared file, resize, save, and orchestration helpers for prompts and media.
hyvideo/utils/data_utils.py
Core model
Text/image conditioning, diffusion backbone, flow scheduler, and VAE.
- I2V pipeline · Central generation pipeline that joins conditioning, latent generation, and decoding.
hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py - Text prep · Boundary from raw prompts to model-ready text tokens and encoder inputs.
hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py - Text encoders · Text encoding package used to produce conditioning embeddings.
hyvideo/text_encoder - Video backbone · Assembly point for the denoising backbone and multimodal generation core.
hyvideo/modules/models.py - Core layers · Attention, MLP, embeddings, modulation, normalization, and token refinement layers.
hyvideo/modules - Flow math · Transport, path, integrator, and utility code for sampling trajectories.
hyvideo/diffusion/flow - Flow scheduler · Discrete scheduler that drives iterative denoising / flow steps during inference.
hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py - Video VAE · Latent-space encoder/decoder for video frames and generated latents.
hyvideo/vae
LoRA training
Adapter fine-tuning and distributed training support.
- Train LoRA · Fine-tuning entrypoint for effect-specific LoRA adapter training.
train_image2video_lora.py - LoRA inject · Loads and fuses LoRA weights into the base model during inference.
hyvideo/utils/lora_utils.py - Train utils · Training helpers for optimization, memory efficiency, and launch setup.
hyvideo/utils/train_utils.py - DS config · Distributed/memory configuration used by the LoRA training path.
hyvideo/ds_config.py
Data prep
Video ingestion and VAE-based preprocessing for training data.
- VAE extract · Standalone preprocessing pipeline that builds VAE-related training artifacts from raw videos.
hyvideo/hyvae_extract/run.py - Extract dataset · Dataset reader used by the VAE extraction workflow.
hyvideo/hyvae_extract/dataset.py - Video loader · Video sample loading for training and preprocessing workflows.
hyvideo/dataset/video_loader.py
Connections 21
- Sample I2V → Inference glue: calls · CLI dispatches to shared inference orchestration.
- Inference glue → I2V pipeline: builds · Inference glue constructs and runs the generation pipeline.
- Run scripts → Sample I2V: launches · Shell wrappers start common sampling modes and multi-GPU runs.
- Train LoRA → Train utils: uses · Training entrypoint relies on helper utilities for execution and optimization.
- Train LoRA → DS config: configures · Training uses distributed and memory-oriented configuration.
- Train LoRA → Video backbone: optimizes · LoRA training targets the shared model backbone.
- Train LoRA → LoRA inject: fits adapters · Training produces adapter weights compatible with inference injection.
- Sample I2V → LoRA inject: loads LoRA · Inference can optionally fuse adapter weights into the base model.
- I2V pipeline → Text prep: tokenizes · Pipeline prepares prompt inputs for the text encoder.
- Text prep → Text encoders: feeds · Tokenizer and prompt utilities prepare inputs for encoding.
- I2V pipeline → Video backbone: runs · The pipeline executes the video denoising backbone.
- Video backbone → Core layers: assembled from · The backbone is built from reusable attention and feed-forward layers.
- I2V pipeline → Flow math: steps through · Sampling trajectory is governed by flow-matching math.
- Flow math → Flow scheduler: drives · Scheduler coordinates the discrete flow-matching update loop.
- I2V pipeline → Video VAE: decodes via · Generated latent video is turned back into pixels through the VAE.
- I2V pipeline → Data utils: uses · Pipeline relies on shared loading and output helpers.
- VAE extract → Extract dataset: reads · Extraction pipeline consumes raw video datasets through its own reader.
- VAE extract → Video VAE: extracts latents · VAE-related preprocessing generates training artifacts from videos.
- Extract dataset → Video loader: builds on · Extraction dataset logic depends on the repository's video ingestion layer.
- Video loader → Data utils: supports · Video loading shares media and filesystem helpers.
- VAE extract → Train LoRA: feeds training · Processed artifacts become inputs for LoRA fine-tuning.