# tencent-hunyuan/hunyuanvideo-i2v architecture

> Architecture diagram of the GitHub repository tencent-hunyuan/hunyuanvideo-i2v, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tencent-hunyuan/hunyuanvideo-i2v
- Repository: https://github.com/tencent-hunyuan/hunyuanvideo-i2v

## Overview

1. **Repo purpose and shape**
   - This is a **PyTorch image-to-video generation system** built around a pretrained HunyuanVideo base model.
   - The repository is split into three clear planes: **inference/sampling**, **LoRA fine-tuning**, and **data preprocessing / VAE latent extraction**.
   - It is **CLI-first**, not a service or web application.

2. **Primary entrypoints**
   - `sample_image2video.py` is the main inference entrypoint for both standard I2V and LoRA-augmented generation.
   - `train_image2video_lora.py` is the fine-tuning entrypoint for effect-specific LoRA training.
   - Shell wrappers in `scripts/` orchestrate common runs, including dynamic sampling, stability mode, LoRA training, and multi-GPU execution.

3. **Core generation pipeline**
   - The central runtime path is `hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py`.
   - This pipeline connects:
     - the **reference image and text prompt**,
     - the **text/image tokenization and embedding stack**,
     - the **video latent generator**,
     - and the **flow-matching sampler / scheduler**.
   - The output is a decoded video saved to disk, typically as an MP4.

4. **Conditioning and multimodal token flow**
   - The README describes the key I2V design: the reference image is converted into **semantic image tokens** by a pretrained multimodal encoder.
   - Those image tokens are **concatenated with video latent tokens** so the transformer can attend across both modalities.
   - This is the main architectural boundary: **conditioning stage** first, then **joint generation over combined image/video tokens**.

5. **Model backbone**
   - `hyvideo/modules/` contains the transformer-style building blocks: attention, MLP, embeddings, modulation, normalization, positional embeddings, token refinement, and activation layers.
   - `hyvideo/modules/models.py` is the likely assembly point for the denoising / video backbone.
   - `hyvideo/modules/fp8_optimization.py` suggests optional low-precision acceleration paths for inference efficiency.

6. **Latent video representation**
   - The VAE subsystem in `hyvideo/vae/` handles **encoding images/video frames into latent space** and **decoding generated latents back to pixels**.
   - `autoencoder_kl_causal_3d.py` and `unet_causal_3d_blocks.py` indicate a **causal 3D VAE** design suited to video temporal structure.
   - This means generation happens in a compressed latent domain, not directly in pixel space.

7. **Sampling, flow matching, and scheduler**
   - `hyvideo/diffusion/flow/` contains the math for the model’s sampling trajectory: transport, paths, integrators, and helpers.
   - `hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py` is the discrete scheduler used during inference.
   - In practice, the scheduler drives iterative denoising / flow steps from noise toward a video latent, with options like `flow-reverse` and `flow-shift` exposed in the CLI.

8. **Text and image preprocessing**
   - `hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py` and the `hyvideo/text_encoder/` package are the boundary between raw text and model-ready tokens.
   - `hyvideo/utils/data_utils.py`, `helpers.py`, and `file_utils.py` support prompt/image loading, resizing, saving, and general orchestration.
   - `hyvideo/constants.py` and `config.py` centralize model and runtime configuration.

9. **LoRA customization path**
   - LoRA support is integrated into the inference path via `hyvideo/utils/lora_utils.py`.
   - The repository’s LoRA workflow is meant for **effect-specific style/motion adaptation**, such as “hair growth” or “embrace”.
   - At runtime, LoRA weights are loaded and fused into the base model with a user-controlled scale, without changing the base checkpoint.

10. **LoRA training subsystem**
   - `train_image2video_lora.py` uses the same base model components but optimizes only the LoRA adapters.
   - `hyvideo/utils/train_utils.py` and `hyvideo/ds_config.py` indicate training support for distributed / memory-efficient setups.
   - Training consumes prepared video-caption pairs and produces LoRA checkpoint artifacts, which are later injected during inference.

11. **Data preprocessing and VAE extraction**
   - `hyvideo/hyvae_extract/` is a separate preprocessing pipeline for building training data.
   - It reads video datasets, runs VAE-related extraction, and emits processed metadata/artifacts used by LoRA training.
   - This is an important boundary: **raw videos → extracted/processed latent training data → LoRA training**.

12. **Dataset loading**
   - `hyvideo/dataset/video_loader.py` is the repository’s data ingestion layer for video samples.
   - It supports the training/preprocessing workflows rather than serving as a generic dataset library.
   - The `assets/demo/` tree provides example images, videos, and training-data layout for the documented workflows.

13. **Multi-GPU inference path**
   - The repo supports xDiT-based parallel inference through `torchrun` and sequence parallelism.
   - This is an **alternate execution mode** of the same inference pipeline, not a separate model.
   - The architecture changes at runtime by splitting latent computation across GPUs, with `ALLOW_RESIZE_FOR_SP=1` enabling size adjustments for divisibility.

14. **Operational dependencies and artifacts**
   - The project depends on **PyTorch, CUDA, flash-attn, and optionally xDiT/xfuser** for speed and memory efficiency.
   - Checkpoints are managed under `ckpts/`, and the README points to Hugging Face / downloadable weights.
   - Generated outputs are written to user-selected result directories; there is no persistent backend service or database layer.

15. **How to draw the main architecture**
   - A useful diagram would show:
     - **Input image + prompt**
     - → **text/image encoding**
     - → **joint tokenization with video latents**
     - → **diffusion/flow transformer backbone**
     - → **scheduler/integrator loop**
     - → **VAE decoder**
     - → **output video**
   - Beside that main lane, show a parallel **LoRA training lane** fed by **preprocessed video datasets** and producing adapter weights that plug back into the same inference pipeline.

Main entry points: [`hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py`](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py) (I2V pipeline), [`train_image2video_lora.py`](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/train_image2video_lora.py) (Train LoRA), [`hyvideo/hyvae_extract/run.py`](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/hyvae_extract/run.py) (VAE extract), [`sample_image2video.py`](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/sample_image2video.py) (Sample I2V)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli_inference["CLI & inference"]
  node_sample_image2video_py["Sample I2V<br/>CLI entrypoint"]
  node_scripts_bundle["Run scripts<br/>shell wrappers"]
  node_hyvideo_inference_py["Inference glue<br/>runtime orchestration<br/>[inference.py]"]
  node_data_utils_py["Data utils<br/>helpers<br/>[data_utils.py]"]
end

subgraph group_core_model["Core model"]
  node_pipeline_hunyuan_video_py["I2V pipeline<br/>diffusion pipeline"]
  node_text_preprocess_utils["Text prep<br/>tokenizer utils"]
  node_text_encoder_pkg["Text encoders<br/>encoder stack"]
  node_modules_models_py["Video backbone<br/>transformer model<br/>[models.py]"]
  node_modules_stack["Core layers<br/>attention/mlp layers"]
  node_flow_stack["Flow math<br/>flow-matching core"]
  node_scheduler_flow_match_py["Flow scheduler<br/>discrete scheduler"]
  node_vae_stack["Video VAE<br/>causal 3D VAE"]
end

subgraph group_lora_training["LoRA training"]
  node_train_image2video_lora_py["Train LoRA<br/>CLI entrypoint"]
  node_lora_utils_py["LoRA inject<br/>adapter utils<br/>[lora_utils.py]"]
  node_train_utils_py["Train utils<br/>distributed training<br/>[train_utils.py]"]
  node_ds_config_py["DS config<br/>training config<br/>[ds_config.py]"]
end

subgraph group_data_prep["Data prep"]
  node_hyvae_extract_run_py["VAE extract<br/>preprocess pipeline<br/>[run.py]"]
  node_hyvae_extract_dataset_py["Extract dataset<br/>dataset stage<br/>[dataset.py]"]
  node_video_loader_py["Video loader<br/>data ingestion<br/>[video_loader.py]"]
end

node_sample_image2video_py -->|"calls"| node_hyvideo_inference_py
node_hyvideo_inference_py -->|"builds"| node_pipeline_hunyuan_video_py
node_scripts_bundle -.->|"launches"| node_sample_image2video_py
node_train_image2video_lora_py -->|"uses"| node_train_utils_py
node_train_image2video_lora_py -->|"configures"| node_ds_config_py
node_train_image2video_lora_py -->|"optimizes"| node_modules_models_py
node_train_image2video_lora_py -->|"fits adapters"| node_lora_utils_py
node_sample_image2video_py -->|"loads LoRA"| node_lora_utils_py
node_pipeline_hunyuan_video_py -->|"tokenizes"| node_text_preprocess_utils
node_text_preprocess_utils -->|"feeds"| node_text_encoder_pkg
node_pipeline_hunyuan_video_py -->|"runs"| node_modules_models_py
node_modules_models_py -->|"assembled from"| node_modules_stack
node_pipeline_hunyuan_video_py -->|"steps through"| node_flow_stack
node_flow_stack -->|"drives"| node_scheduler_flow_match_py
node_pipeline_hunyuan_video_py -->|"decodes via"| node_vae_stack
node_pipeline_hunyuan_video_py -->|"uses"| node_data_utils_py
node_hyvae_extract_run_py -->|"reads"| node_hyvae_extract_dataset_py
node_hyvae_extract_run_py -->|"extracts latents"| node_vae_stack
node_hyvae_extract_dataset_py -->|"builds on"| node_video_loader_py
node_video_loader_py -->|"supports"| node_data_utils_py
node_hyvae_extract_run_py -.->|"feeds training"| node_train_image2video_lora_py

click node_sample_image2video_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/sample_image2video.py"
click node_train_image2video_lora_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/train_image2video_lora.py"
click node_scripts_bundle "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/scripts"
click node_hyvideo_inference_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/inference.py"
click node_pipeline_hunyuan_video_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py"
click node_text_preprocess_utils "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py"
click node_text_encoder_pkg "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/text_encoder"
click node_modules_models_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/modules/models.py"
click node_modules_stack "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/modules"
click node_flow_stack "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/diffusion/flow"
click node_scheduler_flow_match_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py"
click node_vae_stack "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/vae"
click node_lora_utils_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/lora_utils.py"
click node_train_utils_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/train_utils.py"
click node_ds_config_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/ds_config.py"
click node_hyvae_extract_run_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/hyvae_extract/run.py"
click node_hyvae_extract_dataset_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/hyvae_extract/dataset.py"
click node_video_loader_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/dataset/video_loader.py"
click node_data_utils_py "https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/data_utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sample_image2video_py,node_scripts_bundle,node_hyvideo_inference_py,node_data_utils_py toneBlue
class node_pipeline_hunyuan_video_py,node_text_preprocess_utils,node_text_encoder_pkg,node_modules_models_py,node_modules_stack,node_flow_stack,node_scheduler_flow_match_py,node_vae_stack toneAmber
class node_train_image2video_lora_py,node_lora_utils_py,node_train_utils_py,node_ds_config_py toneMint
class node_hyvae_extract_run_py,node_hyvae_extract_dataset_py,node_video_loader_py toneRose
```

## Components

### CLI & inference

User-facing sampling entrypoints and runtime orchestration.

- [Sample I2V](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/sample_image2video.py): CLI entrypoint · Main inference entrypoint for standard and LoRA-augmented image-to-video generation. · `sample_image2video.py`
- [Run scripts](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/scripts): shell wrappers · Shell launchers for sampling, stability mode, LoRA runs, and multi-GPU execution. · `scripts`
- [Inference glue](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/inference.py): runtime orchestration · Shared inference helpers that assemble model loading, prompt handling, and generation execution. · `hyvideo/inference.py`
- [Data utils](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/data_utils.py): helpers · Shared file, resize, save, and orchestration helpers for prompts and media. · `hyvideo/utils/data_utils.py`

### Core model

Text/image conditioning, diffusion backbone, flow scheduler, and VAE.

- [I2V pipeline](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py): diffusion pipeline · Central generation pipeline that joins conditioning, latent generation, and decoding. · `hyvideo/diffusion/pipelines/pipeline_hunyuan_video.py`
- [Text prep](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py): tokenizer utils · Boundary from raw prompts to model-ready text tokens and encoder inputs. · `hyvideo/utils/preprocess_text_encoder_tokenizer_utils.py`
- [Text encoders](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/text_encoder): encoder stack · Text encoding package used to produce conditioning embeddings. · `hyvideo/text_encoder`
- [Video backbone](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/modules/models.py): transformer model · Assembly point for the denoising backbone and multimodal generation core. · `hyvideo/modules/models.py`
- [Core layers](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/modules): attention/mlp layers · Attention, MLP, embeddings, modulation, normalization, and token refinement layers. · `hyvideo/modules`
- [Flow math](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/diffusion/flow): flow-matching core · Transport, path, integrator, and utility code for sampling trajectories. · `hyvideo/diffusion/flow`
- [Flow scheduler](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py): discrete scheduler · Discrete scheduler that drives iterative denoising / flow steps during inference. · `hyvideo/diffusion/schedulers/scheduling_flow_match_discrete.py`
- [Video VAE](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/tree/main/hyvideo/vae): causal 3D VAE · Latent-space encoder/decoder for video frames and generated latents. · `hyvideo/vae`

### LoRA training

Adapter fine-tuning and distributed training support.

- [Train LoRA](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/train_image2video_lora.py): CLI entrypoint · Fine-tuning entrypoint for effect-specific LoRA adapter training. · `train_image2video_lora.py`
- [LoRA inject](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/lora_utils.py): adapter utils · Loads and fuses LoRA weights into the base model during inference. · `hyvideo/utils/lora_utils.py`
- [Train utils](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/utils/train_utils.py): distributed training · Training helpers for optimization, memory efficiency, and launch setup. · `hyvideo/utils/train_utils.py`
- [DS config](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/ds_config.py): training config · Distributed/memory configuration used by the LoRA training path. · `hyvideo/ds_config.py`

### Data prep

Video ingestion and VAE-based preprocessing for training data.

- [VAE extract](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/hyvae_extract/run.py): preprocess pipeline · Standalone preprocessing pipeline that builds VAE-related training artifacts from raw videos. · `hyvideo/hyvae_extract/run.py`
- [Extract dataset](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/hyvae_extract/dataset.py): dataset stage · Dataset reader used by the VAE extraction workflow. · `hyvideo/hyvae_extract/dataset.py`
- [Video loader](https://github.com/tencent-hunyuan/hunyuanvideo-i2v/blob/main/hyvideo/dataset/video_loader.py): data ingestion · Video sample loading for training and preprocessing workflows. · `hyvideo/dataset/video_loader.py`

## Connections

- Sample I2V → Inference glue: calls · CLI dispatches to shared inference orchestration.
- Inference glue → I2V pipeline: builds · Inference glue constructs and runs the generation pipeline.
- Run scripts → Sample I2V: launches · Shell wrappers start common sampling modes and multi-GPU runs.
- Train LoRA → Train utils: uses · Training entrypoint relies on helper utilities for execution and optimization.
- Train LoRA → DS config: configures · Training uses distributed and memory-oriented configuration.
- Train LoRA → Video backbone: optimizes · LoRA training targets the shared model backbone.
- Train LoRA → LoRA inject: fits adapters · Training produces adapter weights compatible with inference injection.
- Sample I2V → LoRA inject: loads LoRA · Inference can optionally fuse adapter weights into the base model.
- I2V pipeline → Text prep: tokenizes · Pipeline prepares prompt inputs for the text encoder.
- Text prep → Text encoders: feeds · Tokenizer and prompt utilities prepare inputs for encoding.
- I2V pipeline → Video backbone: runs · The pipeline executes the video denoising backbone.
- Video backbone → Core layers: assembled from · The backbone is built from reusable attention and feed-forward layers.
- I2V pipeline → Flow math: steps through · Sampling trajectory is governed by flow-matching math.
- Flow math → Flow scheduler: drives · Scheduler coordinates the discrete flow-matching update loop.
- I2V pipeline → Video VAE: decodes via · Generated latent video is turned back into pixels through the VAE.
- I2V pipeline → Data utils: uses · Pipeline relies on shared loading and output helpers.
- VAE extract → Extract dataset: reads · Extraction pipeline consumes raw video datasets through its own reader.
- VAE extract → Video VAE: extracts latents · VAE-related preprocessing generates training artifacts from videos.
- Extract dataset → Video loader: builds on · Extraction dataset logic depends on the repository's video ingestion layer.
- Video loader → Data utils: supports · Video loading shares media and filesystem helpers.
- VAE extract → Train LoRA: feeds training · Processed artifacts become inputs for LoRA fine-tuning.
