# pku-yuangroup/open-sora-plan architecture

> Architecture diagram of the GitHub repository pku-yuangroup/open-sora-plan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pku-yuangroup/open-sora-plan
- Repository: https://github.com/pku-yuangroup/open-sora-plan

## Overview

Open-Sora Plan is a video-generation research system whose principal path is prompt/video data preparation, text conditioning, latent video modeling, and sampled video delivery through a web interface. The repository also contains a separate inpainting path, causal/WF-VAE training and evaluation, prompt refinement, and frame interpolation. Training scripts orchestrate datasets, encoders, autoencoders, and diffusion models; sequence-parallel communication and Ascend configuration provide an internal execution layer. Some sampling pipelines named by the source index are absent from the supplied tree, so those links are omitted.

Main entry points: [`opensora/train/train_t2v_diffusers.py`](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_t2v_diffusers.py) (T2V training), [`opensora/utils/sample_utils.py`](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/utils/sample_utils.py) (Sampling utilities), [`opensora/models/causalvideovae/model/vae/modeling_causalvae.py`](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/causalvideovae/model/vae/modeling_causalvae.py) (Causal and WF-VAE), [`opensora/train/train_inpaint.py`](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_inpaint.py) (Inpaint training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_product["Generation product"]
  node_web["Generation UI"]
  node_web_i2v["I2V UI"]
  node_sampling["Sampling utilities<br/>[sample_utils.py]"]
  node_refiner["Prompt refiner<br/>[inference.py]"]
  node_interpolation["Frame interpolation<br/>[interpolation.py]"]
end

subgraph group_learning["Training and data"]
  node_t2vdata["T2V dataset<br/>[t2v_datasets.py]"]
  node_inpaintdata["Inpaint dataset<br/>[inpaint_dataset.py]"]
  node_transforms["Video transforms<br/>[transform.py]"]
  node_train_t2v["T2V training"]
  node_train_inpaint["Inpaint training<br/>[train_inpaint.py]"]
  node_train_vae["VAE training<br/>[train_causalvae.py]"]
end

subgraph group_models["Video models"]
  node_text["Text encoders<br/>[t5.py]"]
  node_diffusion["Video diffusion"]
  node_inpaint_model["Inpaint diffusion"]
  node_vae["Causal and WF-VAE"]
  node_vae_eval["VAE evaluation<br/>[eval.py]"]
end

subgraph group_runtime["Runtime support"]
  node_accel["Parallel communication<br/>[communications.py]"]
  node_parallel["Parallel state<br/>[parallel_states.py]"]
  node_npu["Ascend configuration<br/>[npu_config.py]"]
  node_engine["Distributed engine<br/>[engine.py]"]
  node_zp["ZeRO groups<br/>[zp_manager.py]"]
end

node_user(("Creator"))

node_user -->|"submits prompt"| node_web
node_user -->|"submits image"| node_web_i2v
node_web -->|"calls sampling"| node_sampling
node_web_i2v -->|"calls sampling"| node_sampling
node_t2vdata -->|"uses transforms"| node_transforms
node_train_t2v -->|"loads data"| node_t2vdata
node_train_t2v -->|"uses encoders"| node_text
node_train_t2v -->|"uses autoencoder"| node_vae
node_train_t2v -->|"trains model"| node_diffusion
node_train_t2v -->|"uses collectives"| node_accel
node_train_t2v -->|"configures runtime"| node_npu
node_train_inpaint -->|"loads data"| node_inpaintdata
node_train_inpaint -->|"trains model"| node_inpaint_model
node_train_inpaint -->|"uses autoencoder"| node_vae
node_train_vae -->|"trains autoencoder"| node_vae
node_sampling -->|"uses model"| node_diffusion
node_sampling -->|"uses model"| node_inpaint_model
node_sampling -->|"uses autoencoder"| node_vae
node_train_vae -.->|"evaluates outputs"| node_vae_eval
node_interpolation -.->|"produces frames"| node_user
node_parallel -->|"configures groups"| node_accel
node_npu -->|"uses manager"| node_zp
node_engine -->|"uses manager"| node_zp
node_refiner -.->|"refines prompts"| node_user

click node_web "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/serve/gradio_web_server.py"
click node_web_i2v "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/serve/gradio_web_server_i2v.py"
click node_sampling "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/utils/sample_utils.py"
click node_refiner "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/prompt_refiner/inference.py"
click node_interpolation "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/frame_interpolation/interpolation.py"
click node_t2vdata "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/dataset/t2v_datasets.py"
click node_inpaintdata "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/dataset/inpaint_dataset.py"
click node_transforms "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/dataset/transform.py"
click node_train_t2v "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_t2v_diffusers.py"
click node_train_inpaint "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_inpaint.py"
click node_train_vae "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_causalvae.py"
click node_text "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/text_encoder/t5.py"
click node_diffusion "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/diffusion/opensora_v1_3/modeling_opensora.py"
click node_inpaint_model "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/diffusion/opensora_v1_3/modeling_inpaint.py"
click node_vae "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/causalvideovae/model/vae/modeling_causalvae.py"
click node_vae_eval "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/causalvideovae/eval/eval.py"
click node_accel "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/acceleration/communications.py"
click node_parallel "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/acceleration/parallel_states.py"
click node_npu "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/npu_config.py"
click node_engine "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/adaptor/engine.py"
click node_zp "https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/adaptor/zp_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web,node_web_i2v,node_sampling,node_refiner,node_interpolation toneBlue
class node_t2vdata,node_inpaintdata,node_transforms,node_train_t2v,node_train_inpaint,node_train_vae toneAmber
class node_text,node_diffusion,node_inpaint_model,node_vae,node_vae_eval toneMint
class node_accel,node_parallel,node_npu,node_engine,node_zp toneRose
class node_user toneIndigo
```

## Components

### Generation product

- [Generation UI](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/serve/gradio_web_server.py): `opensora/serve/gradio_web_server.py`
- [I2V UI](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/serve/gradio_web_server_i2v.py): `opensora/serve/gradio_web_server_i2v.py`
- [Sampling utilities](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/utils/sample_utils.py): `opensora/utils/sample_utils.py`
- [Prompt refiner](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/prompt_refiner/inference.py): `opensora/models/prompt_refiner/inference.py`
- [Frame interpolation](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/frame_interpolation/interpolation.py): `opensora/models/frame_interpolation/interpolation.py`

### Training and data

- [T2V dataset](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/dataset/t2v_datasets.py): `opensora/dataset/t2v_datasets.py`
- [Inpaint dataset](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/dataset/inpaint_dataset.py): `opensora/dataset/inpaint_dataset.py`
- [Video transforms](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/dataset/transform.py): `opensora/dataset/transform.py`
- [T2V training](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_t2v_diffusers.py): `opensora/train/train_t2v_diffusers.py`
- [Inpaint training](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_inpaint.py): `opensora/train/train_inpaint.py`
- [VAE training](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/train/train_causalvae.py): `opensora/train/train_causalvae.py`

### Video models

- [Text encoders](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/text_encoder/t5.py): `opensora/models/text_encoder/t5.py`
- [Video diffusion](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/diffusion/opensora_v1_3/modeling_opensora.py): `opensora/models/diffusion/opensora_v1_3/modeling_opensora.py`
- [Inpaint diffusion](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/diffusion/opensora_v1_3/modeling_inpaint.py): `opensora/models/diffusion/opensora_v1_3/modeling_inpaint.py`
- [Causal and WF-VAE](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/causalvideovae/model/vae/modeling_causalvae.py): `opensora/models/causalvideovae/model/vae/modeling_causalvae.py`
- [VAE evaluation](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/models/causalvideovae/eval/eval.py): `opensora/models/causalvideovae/eval/eval.py`

### Runtime support

- [Parallel communication](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/acceleration/communications.py): `opensora/acceleration/communications.py`
- [Parallel state](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/acceleration/parallel_states.py): `opensora/acceleration/parallel_states.py`
- [Ascend configuration](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/npu_config.py): `opensora/npu_config.py`
- [Distributed engine](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/adaptor/engine.py): `opensora/adaptor/engine.py`
- [ZeRO groups](https://github.com/pku-yuangroup/open-sora-plan/blob/main/opensora/adaptor/zp_manager.py): `opensora/adaptor/zp_manager.py`

### Other components

- **Creator**

## Connections

- Creator → Generation UI: submits prompt (evidence: `opensora/serve/gradio_web_server.py`)
- Creator → I2V UI: submits image (evidence: `opensora/serve/gradio_web_server_i2v.py`)
- Generation UI → Sampling utilities: calls sampling (evidence: `opensora/serve/gradio_web_server.py`)
- I2V UI → Sampling utilities: calls sampling (evidence: `opensora/serve/gradio_web_server_i2v.py`)
- T2V dataset → Video transforms: uses transforms (evidence: `opensora/dataset/t2v_datasets.py`)
- T2V training → T2V dataset: loads data (evidence: `opensora/train/train_t2v_diffusers.py`)
- T2V training → Text encoders: uses encoders (evidence: `opensora/train/train_t2v_diffusers.py`)
- T2V training → Causal and WF-VAE: uses autoencoder (evidence: `opensora/train/train_t2v_diffusers.py`)
- T2V training → Video diffusion: trains model (evidence: `opensora/train/train_t2v_diffusers.py`)
- T2V training → Parallel communication: uses collectives (evidence: `opensora/train/train_t2v_diffusers.py`)
- T2V training → Ascend configuration: configures runtime (evidence: `opensora/train/train_t2v_diffusers.py`)
- Inpaint training → Inpaint dataset: loads data (evidence: `opensora/train/train_inpaint.py`)
- Inpaint training → Inpaint diffusion: trains model (evidence: `opensora/train/train_inpaint.py`)
- Inpaint training → Causal and WF-VAE: uses autoencoder (evidence: `opensora/train/train_inpaint.py`)
- VAE training → Causal and WF-VAE: trains autoencoder (evidence: `opensora/train/train_causalvae.py`)
- Sampling utilities → Video diffusion: uses model (evidence: `opensora/utils/sample_utils.py`)
- Sampling utilities → Inpaint diffusion: uses model (evidence: `opensora/utils/sample_utils.py`)
- Sampling utilities → Causal and WF-VAE: uses autoencoder (evidence: `opensora/utils/sample_utils.py`)
- VAE training → VAE evaluation: evaluates outputs
- Frame interpolation → Creator: produces frames
- Parallel state → Parallel communication: configures groups (evidence: `opensora/acceleration/communications.py`)
- Ascend configuration → ZeRO groups: uses manager (evidence: `opensora/npu_config.py`)
- Distributed engine → ZeRO groups: uses manager (evidence: `opensora/adaptor/engine.py`)
- Prompt refiner → Creator: refines prompts
