How pku-yuangroup/open-sora-plan works
Open-Sora Plan is a video-generation research system whose principal path is prompt/video data preparation, text conditioning, latent video modeling, and sampled video delivery through a web interface. The repository also contains a separate inpainting path, causal/WF-VAE training and evaluation, prompt refinement, and frame interpolation. Training scripts orchestrate datasets, encoders, autoencoders, and diffusion models; sequence-parallel communication and Ascend configuration provide an internal execution layer. Some sampling pipelines named by the source index are absent from the supplied tree, so those links are omitted.
Main entry points: train_t2v_diffusers.py T2V training · sample_utils.py Sampling utilities · modeling_causalvae.py Causal and WF-VAE · train_inpaint.py Inpaint training
Components 22
Generation product
- Generation UI
opensora/serve/gradio_web_server.py - I2V UI
opensora/serve/gradio_web_server_i2v.py - Sampling utilities
opensora/utils/sample_utils.py - Prompt refiner
opensora/models/prompt_refiner/inference.py - Frame interpolation
opensora/models/frame_interpolation/interpolation.py
Training and data
- T2V dataset
opensora/dataset/t2v_datasets.py - Inpaint dataset
opensora/dataset/inpaint_dataset.py - Video transforms
opensora/dataset/transform.py - T2V training
opensora/train/train_t2v_diffusers.py - Inpaint training
opensora/train/train_inpaint.py - VAE training
opensora/train/train_causalvae.py
Video models
- Text encoders
opensora/models/text_encoder/t5.py - Video diffusion
opensora/models/diffusion/opensora_v1_3/modeling_opensora.py - Inpaint diffusion
opensora/models/diffusion/opensora_v1_3/modeling_inpaint.py - Causal and WF-VAE
opensora/models/causalvideovae/model/vae/modeling_causalvae.py - VAE evaluation
opensora/models/causalvideovae/eval/eval.py
Runtime support
- Parallel communication
opensora/acceleration/communications.py - Parallel state
opensora/acceleration/parallel_states.py - Ascend configuration
opensora/npu_config.py - Distributed engine
opensora/adaptor/engine.py - ZeRO groups
opensora/adaptor/zp_manager.py
Other components
- Creator
Connections 24
- Creator → Generation UI: submits prompt
opensora/serve/gradio_web_server.py - Creator → I2V UI: submits image
opensora/serve/gradio_web_server_i2v.py - Generation UI → Sampling utilities: calls sampling
opensora/serve/gradio_web_server.py - I2V UI → Sampling utilities: calls sampling
opensora/serve/gradio_web_server_i2v.py - T2V dataset → Video transforms: uses transforms
opensora/dataset/t2v_datasets.py - T2V training → T2V dataset: loads data
opensora/train/train_t2v_diffusers.py - T2V training → Text encoders: uses encoders
opensora/train/train_t2v_diffusers.py - T2V training → Causal and WF-VAE: uses autoencoder
opensora/train/train_t2v_diffusers.py - T2V training → Video diffusion: trains model
opensora/train/train_t2v_diffusers.py - T2V training → Parallel communication: uses collectives
opensora/train/train_t2v_diffusers.py - T2V training → Ascend configuration: configures runtime
opensora/train/train_t2v_diffusers.py - Inpaint training → Inpaint dataset: loads data
opensora/train/train_inpaint.py - Inpaint training → Inpaint diffusion: trains model
opensora/train/train_inpaint.py - Inpaint training → Causal and WF-VAE: uses autoencoder
opensora/train/train_inpaint.py - VAE training → Causal and WF-VAE: trains autoencoder
opensora/train/train_causalvae.py - Sampling utilities → Video diffusion: uses model
opensora/utils/sample_utils.py - Sampling utilities → Inpaint diffusion: uses model
opensora/utils/sample_utils.py - Sampling utilities → Causal and WF-VAE: uses autoencoder
opensora/utils/sample_utils.py - VAE training → VAE evaluation: evaluates outputs
- Frame interpolation → Creator: produces frames
- Parallel state → Parallel communication: configures groups
opensora/acceleration/communications.py - Ascend configuration → ZeRO groups: uses manager
opensora/npu_config.py - Distributed engine → ZeRO groups: uses manager
opensora/adaptor/engine.py - Prompt refiner → Creator: refines prompts