How stability-ai/generative-models works
This repository provides configurable generative-model components and sampling workflows, with the README emphasizing image-to-video SV3D and video-to-4D SV4D/SV4D 2.0. Users supply an image or video to a sampling script; the documented result is generated novel-view video, saved output, and (for SV4D 2.0) autoregressive extension for longer clips. The graph includes the diffusion, conditioning, temporal-model, and sampling layers. Some script-to-runtime links are dashed because the sampled excerpts do not show their full inference calls.
Main entry points: diffusion.py Diffusion engine · simple_video_sample.py SV3D sampling · temporal_ae.py Temporal autoencoding · video_model.py Video UNet
Components 22
Sampling workflows
- SV3D sampling
scripts/sampling/simple_video_sample.py - SV4D sampling
scripts/sampling/simple_video_sample_4d.py - SV4D 2.0 sampling
scripts/sampling/simple_video_sample_4d2.py - Inference API
sgm/inference/api.py - Inference helpers
sgm/inference/helpers.py - Safety detection
scripts/util/detection/nsfw_and_watermark_dectection.py
Diffusion engine
- Diffusion engine
sgm/models/diffusion.py - Autoencoder engine
sgm/models/autoencoder.py - Diffusion sampler
sgm/modules/diffusionmodules/sampling.py - Conditioning encoders
sgm/modules/encoders/modules.py
Model networks
- Image UNet
sgm/modules/diffusionmodules/openaimodel.py - Video UNet
sgm/modules/diffusionmodules/video_model.py - Image autoencoder network
sgm/modules/diffusionmodules/model.py - Temporal autoencoding
sgm/modules/autoencoding/temporal_ae.py - Attention layers
sgm/modules/attention.py - Video attention
sgm/modules/video_attention.py - Space-time attention
sgm/modules/spacetime_attention.py
Model support
- Latent regularizers
sgm/modules/autoencoding/regularizers/base.py - Latent distributions
sgm/modules/distributions/distributions.py - EMA weights
sgm/modules/ema.py - Perceptual metric
sgm/modules/autoencoding/lpips/loss/lpips.py
Other components
- Research user
Connections 28
- Research user → SV3D sampling: supplies image
- Research user → SV4D sampling: supplies video
- Research user → SV4D 2.0 sampling: supplies video
- SV3D sampling → Inference helpers: uses helpers
scripts/sampling/simple_video_sample.py - SV3D sampling → Safety detection: imports detector
scripts/sampling/simple_video_sample.py - SV3D sampling → Diffusion engine: runs inference
- SV4D sampling → Diffusion engine: runs inference
- SV4D 2.0 sampling → Diffusion engine: runs inference
- SV4D sampling → SV3D sampling: generates reference views
- Inference API → Inference helpers: uses helpers
sgm/inference/api.py - Inference API → Diffusion sampler: calls sampler
sgm/inference/api.py - Inference helpers → Diffusion engine: denoises and decodes
sgm/inference/helpers.py - Diffusion engine → Temporal autoencoding: configures temporal stage
sgm/models/diffusion.py - Diffusion engine → EMA weights: tracks model weights
sgm/models/diffusion.py - Autoencoder engine → Latent regularizers: configures regularizer
sgm/models/autoencoder.py - Autoencoder engine → EMA weights: uses EMA
sgm/models/autoencoder.py - Temporal autoencoding → Image autoencoder network: uses network blocks
sgm/modules/autoencoding/temporal_ae.py - Temporal autoencoding → Image UNet: uses network blocks
sgm/modules/autoencoding/temporal_ae.py - Temporal autoencoding → Video attention: uses attention
sgm/modules/autoencoding/temporal_ae.py - Video UNet → Image UNet: extends UNet blocks
sgm/modules/diffusionmodules/video_model.py - Video UNet → Video attention: uses attention
sgm/modules/diffusionmodules/video_model.py - Video UNet → Space-time attention: uses attention
sgm/modules/diffusionmodules/video_model.py - Image UNet → Attention layers: uses attention
sgm/modules/diffusionmodules/openaimodel.py - Video attention → Attention layers: uses layers
sgm/modules/video_attention.py - Space-time attention → Attention layers: uses layers
sgm/modules/spacetime_attention.py - Conditioning encoders → Latent regularizers: uses regularizers
sgm/modules/encoders/modules.py - Conditioning encoders → Latent distributions: uses distributions
sgm/modules/encoders/modules.py - Latent regularizers → Latent distributions: uses distributions
sgm/modules/autoencoding/regularizers/__init__.py