# hpcaitech/open-sora architecture

> Architecture diagram of the GitHub repository hpcaitech/open-sora, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hpcaitech/open-sora
- Repository: https://github.com/hpcaitech/open-sora

## Overview

Open-Sora is an open video-generation system with a Gradio interface and command-line workflows for inference and training. The principal generation path accepts prompts and optional image conditioning, prepares text and latent inputs, denoises with a diffusion model, decodes with a video autoencoder, then saves generated media. The repository also supports video-data preparation and model/VAE training. The source excerpts and index substantiate core model, data, sampling, and support layers; several script-level connections are documented only as capabilities, so those links are dashed where unverified.

Main entry points: [`opensora/utils/sampling.py`](https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/sampling.py) (Diffusion sampling), [`scripts/vae/train.py`](https://github.com/hpcaitech/open-sora/blob/main/scripts/vae/train.py) (VAE training), [`opensora/datasets/datasets.py`](https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/datasets.py) (Video datasets), [`gradio/app.py`](https://github.com/hpcaitech/open-sora/blob/main/gradio/app.py) (Gradio demo)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Generation interface"]
  node_demo["Gradio demo<br/>[app.py]"]
  node_inference_cli["Inference runner<br/>[inference.py]"]
end

subgraph group_generation["Generation runtime"]
  node_gen_api["Generation setup<br/>[sampling.py]"]
  node_prompt["Prompt refinement<br/>[prompt_refine.py]"]
  node_denoise["Diffusion sampling<br/>[sampling.py]"]
  node_output["Media output<br/>[inference.py]"]
  node_registry["Component registry<br/>[registry.py]"]
end

subgraph group_models["Model components"]
  node_stdit["MMDiT model<br/>[model.py]"]
  node_mmdit_layers["Transformer layers<br/>[layers.py]"]
  node_mmdit_math["Attention math<br/>[math.py]"]
  node_text["Text conditioner<br/>[conditioner.py]"]
  node_hvae["Hunyuan video VAE"]
  node_dc_ae["DC autoencoder<br/>[dc_ae.py]"]
  node_dc_ops["DC-AE operators<br/>[ops.py]"]
end

subgraph group_data["Video data"]
  node_dataset["Video datasets<br/>[datasets.py]"]
  node_video_read["Video decoding<br/>[read_video.py]"]
  node_transforms["Video transforms"]
  node_aspect["Aspect sizing<br/>[aspect.py]"]
  node_sampler["Bucket sampling<br/>[sampler.py]"]
  node_latent_store["Latent and media I/O<br/>[utils.py]"]
end

subgraph group_training["Training and acceleration"]
  node_vae_train["VAE training<br/>[train.py]"]
  node_diff_train["Diffusion training<br/>[train.py]"]
  node_train_utils["Training utilities<br/>[train.py]"]
  node_checkpoint["Activation checkpointing<br/>[checkpoint.py]"]
end

node_user(("Creator"))

node_user -->|"submits prompt"| node_demo
node_demo -->|"loads model"| node_stdit
node_demo -->|"samples video"| node_denoise
node_denoise -->|"prepares models"| node_gen_api
node_denoise -->|"encodes prompts"| node_text
node_denoise -->|"denoises latents"| node_stdit
node_stdit -->|"uses layers"| node_mmdit_layers
node_mmdit_layers -.->|"uses attention math"| node_mmdit_math
node_gen_api -->|"builds modules"| node_registry
node_denoise -.->|"decodes latents"| node_hvae
node_denoise -.->|"passes samples"| node_output
node_output -->|"saves media"| node_latent_store
node_user -.->|"runs generation"| node_inference_cli
node_inference_cli -.->|"invokes sampling"| node_denoise
node_dataset -->|"decodes clips"| node_video_read
node_dataset -.->|"applies transforms"| node_transforms
node_sampler -->|"groups samples"| node_dataset
node_sampler -->|"buckets by aspect"| node_aspect
node_dataset -.->|"uses dimensions"| node_aspect
node_user -.->|"starts training"| node_vae_train
node_user -.->|"starts training"| node_diff_train
node_vae_train -.->|"uses utilities"| node_train_utils
node_diff_train -.->|"uses utilities"| node_train_utils
node_train_utils -.->|"enables checkpointing"| node_checkpoint
node_dc_ae -->|"uses operators"| node_dc_ops
node_vae_train -.->|"trains autoencoder"| node_dc_ae
node_vae_train -.->|"trains VAE"| node_hvae

click node_demo "https://github.com/hpcaitech/open-sora/blob/main/gradio/app.py"
click node_inference_cli "https://github.com/hpcaitech/open-sora/blob/main/scripts/diffusion/inference.py"
click node_gen_api "https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/sampling.py"
click node_prompt "https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/prompt_refine.py"
click node_denoise "https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/sampling.py"
click node_output "https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/inference.py"
click node_stdit "https://github.com/hpcaitech/open-sora/blob/main/opensora/models/mmdit/model.py"
click node_mmdit_layers "https://github.com/hpcaitech/open-sora/blob/main/opensora/models/mmdit/layers.py"
click node_mmdit_math "https://github.com/hpcaitech/open-sora/blob/main/opensora/models/mmdit/math.py"
click node_text "https://github.com/hpcaitech/open-sora/blob/main/opensora/models/text/conditioner.py"
click node_hvae "https://github.com/hpcaitech/open-sora/blob/main/opensora/models/hunyuan_vae/autoencoder_kl_causal_3d.py"
click node_dc_ae "https://github.com/hpcaitech/open-sora/blob/main/opensora/models/dc_ae/models/dc_ae.py"
click node_dc_ops "https://github.com/hpcaitech/open-sora/blob/main/opensora/models/dc_ae/models/nn/ops.py"
click node_dataset "https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/datasets.py"
click node_video_read "https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/read_video.py"
click node_transforms "https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/video_transforms.py"
click node_aspect "https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/aspect.py"
click node_sampler "https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/sampler.py"
click node_vae_train "https://github.com/hpcaitech/open-sora/blob/main/scripts/vae/train.py"
click node_diff_train "https://github.com/hpcaitech/open-sora/blob/main/scripts/diffusion/train.py"
click node_train_utils "https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/train.py"
click node_checkpoint "https://github.com/hpcaitech/open-sora/blob/main/opensora/acceleration/checkpoint.py"
click node_registry "https://github.com/hpcaitech/open-sora/blob/main/opensora/registry.py"
click node_latent_store "https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_demo,node_inference_cli toneBlue
class node_gen_api,node_prompt,node_denoise,node_output,node_registry toneAmber
class node_stdit,node_mmdit_layers,node_mmdit_math,node_text,node_hvae,node_dc_ae,node_dc_ops toneMint
class node_dataset,node_video_read,node_transforms,node_aspect,node_sampler,node_latent_store toneRose
class node_vae_train,node_diff_train,node_train_utils,node_checkpoint,node_user toneIndigo
```

## Components

### Generation interface

- [Gradio demo](https://github.com/hpcaitech/open-sora/blob/main/gradio/app.py): `gradio/app.py`
- [Inference runner](https://github.com/hpcaitech/open-sora/blob/main/scripts/diffusion/inference.py): `scripts/diffusion/inference.py`

### Generation runtime

- [Generation setup](https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/sampling.py): `opensora/utils/sampling.py`
- [Prompt refinement](https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/prompt_refine.py): `opensora/utils/prompt_refine.py`
- [Diffusion sampling](https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/sampling.py): `opensora/utils/sampling.py`
- [Media output](https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/inference.py): `opensora/utils/inference.py`
- [Component registry](https://github.com/hpcaitech/open-sora/blob/main/opensora/registry.py): `opensora/registry.py`

### Model components

- [MMDiT model](https://github.com/hpcaitech/open-sora/blob/main/opensora/models/mmdit/model.py): `opensora/models/mmdit/model.py`
- [Transformer layers](https://github.com/hpcaitech/open-sora/blob/main/opensora/models/mmdit/layers.py): `opensora/models/mmdit/layers.py`
- [Attention math](https://github.com/hpcaitech/open-sora/blob/main/opensora/models/mmdit/math.py): `opensora/models/mmdit/math.py`
- [Text conditioner](https://github.com/hpcaitech/open-sora/blob/main/opensora/models/text/conditioner.py): `opensora/models/text/conditioner.py`
- [Hunyuan video VAE](https://github.com/hpcaitech/open-sora/blob/main/opensora/models/hunyuan_vae/autoencoder_kl_causal_3d.py): `opensora/models/hunyuan_vae/autoencoder_kl_causal_3d.py`
- [DC autoencoder](https://github.com/hpcaitech/open-sora/blob/main/opensora/models/dc_ae/models/dc_ae.py): `opensora/models/dc_ae/models/dc_ae.py`
- [DC-AE operators](https://github.com/hpcaitech/open-sora/blob/main/opensora/models/dc_ae/models/nn/ops.py): `opensora/models/dc_ae/models/nn/ops.py`

### Video data

- [Video datasets](https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/datasets.py): `opensora/datasets/datasets.py`
- [Video decoding](https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/read_video.py): `opensora/datasets/read_video.py`
- [Video transforms](https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/video_transforms.py): `opensora/datasets/video_transforms.py`
- [Aspect sizing](https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/aspect.py): `opensora/datasets/aspect.py`
- [Bucket sampling](https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/sampler.py): `opensora/datasets/sampler.py`
- [Latent and media I/O](https://github.com/hpcaitech/open-sora/blob/main/opensora/datasets/utils.py): `opensora/datasets/utils.py`

### Training and acceleration

- [VAE training](https://github.com/hpcaitech/open-sora/blob/main/scripts/vae/train.py): `scripts/vae/train.py`
- [Diffusion training](https://github.com/hpcaitech/open-sora/blob/main/scripts/diffusion/train.py): `scripts/diffusion/train.py`
- [Training utilities](https://github.com/hpcaitech/open-sora/blob/main/opensora/utils/train.py): `opensora/utils/train.py`
- [Activation checkpointing](https://github.com/hpcaitech/open-sora/blob/main/opensora/acceleration/checkpoint.py): `opensora/acceleration/checkpoint.py`

### Other components

- **Creator**

## Connections

- Creator → Gradio demo: submits prompt (evidence: `gradio/app.py`)
- Gradio demo → MMDiT model: loads model (evidence: `gradio/app.py`)
- Gradio demo → Diffusion sampling: samples video (evidence: `gradio/app.py`)
- Diffusion sampling → Generation setup: prepares models (evidence: `opensora/utils/sampling.py`)
- Diffusion sampling → Text conditioner: encodes prompts (evidence: `opensora/utils/sampling.py`)
- Diffusion sampling → MMDiT model: denoises latents (evidence: `opensora/utils/sampling.py`)
- MMDiT model → Transformer layers: uses layers (evidence: `opensora/models/mmdit/model.py`)
- Transformer layers → Attention math: uses attention math
- Generation setup → Component registry: builds modules (evidence: `opensora/registry.py`)
- Diffusion sampling → Hunyuan video VAE: decodes latents
- Diffusion sampling → Media output: passes samples (evidence: `opensora/utils/sampling.py`)
- Media output → Latent and media I/O: saves media (evidence: `opensora/utils/inference.py`)
- Creator → Inference runner: runs generation
- Inference runner → Diffusion sampling: invokes sampling
- Video datasets → Video decoding: decodes clips (evidence: `opensora/datasets/datasets.py`)
- Video datasets → Video transforms: applies transforms
- Bucket sampling → Video datasets: groups samples (evidence: `opensora/datasets/sampler.py`)
- Bucket sampling → Aspect sizing: buckets by aspect (evidence: `opensora/datasets/sampler.py`)
- Video datasets → Aspect sizing: uses dimensions
- Creator → VAE training: starts training
- Creator → Diffusion training: starts training
- VAE training → Training utilities: uses utilities
- Diffusion training → Training utilities: uses utilities
- Training utilities → Activation checkpointing: enables checkpointing
- DC autoencoder → DC-AE operators: uses operators (evidence: `opensora/models/dc_ae/models/dc_ae.py`)
- VAE training → DC autoencoder: trains autoencoder
- VAE training → Hunyuan video VAE: trains VAE
