How hpcaitech/open-sora works
Open-Sora is an open video-generation system with a Gradio interface and command-line workflows for inference and training. The principal generation path accepts prompts and optional image conditioning, prepares text and latent inputs, denoises with a diffusion model, decodes with a video autoencoder, then saves generated media. The repository also supports video-data preparation and model/VAE training. The source excerpts and index substantiate core model, data, sampling, and support layers; several script-level connections are documented only as capabilities, so those links are dashed where unverified.
Main entry points: sampling.py Diffusion sampling · train.py VAE training · datasets.py Video datasets · app.py Gradio demo
Components 25
Generation interface
- Gradio demo
gradio/app.py - Inference runner
scripts/diffusion/inference.py
Generation runtime
- Generation setup
opensora/utils/sampling.py - Prompt refinement
opensora/utils/prompt_refine.py - Diffusion sampling
opensora/utils/sampling.py - Media output
opensora/utils/inference.py - Component registry
opensora/registry.py
Model components
- MMDiT model
opensora/models/mmdit/model.py - Transformer layers
opensora/models/mmdit/layers.py - Attention math
opensora/models/mmdit/math.py - Text conditioner
opensora/models/text/conditioner.py - Hunyuan video VAE
opensora/models/hunyuan_vae/autoencoder_kl_causal_3d.py - DC autoencoder
opensora/models/dc_ae/models/dc_ae.py - DC-AE operators
opensora/models/dc_ae/models/nn/ops.py
Video data
- Video datasets
opensora/datasets/datasets.py - Video decoding
opensora/datasets/read_video.py - Video transforms
opensora/datasets/video_transforms.py - Aspect sizing
opensora/datasets/aspect.py - Bucket sampling
opensora/datasets/sampler.py - Latent and media I/O
opensora/datasets/utils.py
Training and acceleration
- VAE training
scripts/vae/train.py - Diffusion training
scripts/diffusion/train.py - Training utilities
opensora/utils/train.py - Activation checkpointing
opensora/acceleration/checkpoint.py
Other components
- Creator
Connections 27
- Creator → Gradio demo: submits prompt
gradio/app.py - Gradio demo → MMDiT model: loads model
gradio/app.py - Gradio demo → Diffusion sampling: samples video
gradio/app.py - Diffusion sampling → Generation setup: prepares models
opensora/utils/sampling.py - Diffusion sampling → Text conditioner: encodes prompts
opensora/utils/sampling.py - Diffusion sampling → MMDiT model: denoises latents
opensora/utils/sampling.py - MMDiT model → Transformer layers: uses layers
opensora/models/mmdit/model.py - Transformer layers → Attention math: uses attention math
- Generation setup → Component registry: builds modules
opensora/registry.py - Diffusion sampling → Hunyuan video VAE: decodes latents
- Diffusion sampling → Media output: passes samples
opensora/utils/sampling.py - Media output → Latent and media I/O: saves media
opensora/utils/inference.py - Creator → Inference runner: runs generation
- Inference runner → Diffusion sampling: invokes sampling
- Video datasets → Video decoding: decodes clips
opensora/datasets/datasets.py - Video datasets → Video transforms: applies transforms
- Bucket sampling → Video datasets: groups samples
opensora/datasets/sampler.py - Bucket sampling → Aspect sizing: buckets by aspect
opensora/datasets/sampler.py - Video datasets → Aspect sizing: uses dimensions
- Creator → VAE training: starts training
- Creator → Diffusion training: starts training
- VAE training → Training utilities: uses utilities
- Diffusion training → Training utilities: uses utilities
- Training utilities → Activation checkpointing: enables checkpointing
- DC autoencoder → DC-AE operators: uses operators
opensora/models/dc_ae/models/dc_ae.py - VAE training → DC autoencoder: trains autoencoder
- VAE training → Hunyuan video VAE: trains VAE