# stability-ai/generative-models architecture

> Architecture diagram of the GitHub repository stability-ai/generative-models, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stability-ai/generative-models
- Repository: https://github.com/stability-ai/generative-models

## Overview

This repository provides configurable generative-model components and sampling workflows, with the README emphasizing image-to-video SV3D and video-to-4D SV4D/SV4D 2.0. Users supply an image or video to a sampling script; the documented result is generated novel-view video, saved output, and (for SV4D 2.0) autoregressive extension for longer clips. The graph includes the diffusion, conditioning, temporal-model, and sampling layers. Some script-to-runtime links are dashed because the sampled excerpts do not show their full inference calls.

Main entry points: [`sgm/models/diffusion.py`](https://github.com/stability-ai/generative-models/blob/main/sgm/models/diffusion.py) (Diffusion engine), [`scripts/sampling/simple_video_sample.py`](https://github.com/stability-ai/generative-models/blob/main/scripts/sampling/simple_video_sample.py) (SV3D sampling), [`sgm/modules/autoencoding/temporal_ae.py`](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/autoencoding/temporal_ae.py) (Temporal autoencoding), [`sgm/modules/diffusionmodules/video_model.py`](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/video_model.py) (Video UNet)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sampling["Sampling workflows"]
  node_sv3d["SV3D sampling"]
  node_sv4d["SV4D sampling"]
  node_sv4d2["SV4D 2.0 sampling"]
  node_inference_api["Inference API<br/>[api.py]"]
  node_inference_helpers["Inference helpers<br/>[helpers.py]"]
  node_safety_filter["Safety detection"]
end

subgraph group_engine["Diffusion engine"]
  node_diffusion_engine["Diffusion engine<br/>[diffusion.py]"]
  node_autoencoder["Autoencoder engine<br/>[autoencoder.py]"]
  node_sampler["Diffusion sampler<br/>[sampling.py]"]
  node_conditioning["Conditioning encoders<br/>[modules.py]"]
end

subgraph group_networks["Model networks"]
  node_image_unet["Image UNet<br/>[openaimodel.py]"]
  node_video_unet["Video UNet<br/>[video_model.py]"]
  node_image_network["Image autoencoder network<br/>[model.py]"]
  node_temporal_ae["Temporal autoencoding<br/>[temporal_ae.py]"]
  node_attention["Attention layers<br/>[attention.py]"]
  node_video_attention["Video attention<br/>[video_attention.py]"]
  node_spacetime_attention["Space-time attention"]
end

subgraph group_support["Model support"]
  node_regularizers["Latent regularizers<br/>[base.py]"]
  node_distributions["Latent distributions<br/>[distributions.py]"]
  node_ema["EMA weights<br/>[ema.py]"]
  node_lpips["Perceptual metric<br/>[lpips.py]"]
end

node_user(("Research user"))

node_user -->|"supplies image"| node_sv3d
node_user -->|"supplies video"| node_sv4d
node_user -->|"supplies video"| node_sv4d2
node_sv3d -->|"uses helpers"| node_inference_helpers
node_sv3d -->|"imports detector"| node_safety_filter
node_sv3d -.->|"runs inference"| node_diffusion_engine
node_sv4d -.->|"runs inference"| node_diffusion_engine
node_sv4d2 -.->|"runs inference"| node_diffusion_engine
node_sv4d -.->|"generates reference views"| node_sv3d
node_inference_api -->|"uses helpers"| node_inference_helpers
node_inference_api -->|"calls sampler"| node_sampler
node_inference_helpers -->|"denoises and decodes"| node_diffusion_engine
node_diffusion_engine -->|"configures temporal stage"| node_temporal_ae
node_diffusion_engine -->|"tracks model weights"| node_ema
node_autoencoder -->|"configures regularizer"| node_regularizers
node_autoencoder -->|"uses EMA"| node_ema
node_temporal_ae -->|"uses network blocks"| node_image_network
node_temporal_ae -->|"uses network blocks"| node_image_unet
node_temporal_ae -->|"uses attention"| node_video_attention
node_video_unet -->|"extends UNet blocks"| node_image_unet
node_video_unet -->|"uses attention"| node_video_attention
node_video_unet -->|"uses attention"| node_spacetime_attention
node_image_unet -->|"uses attention"| node_attention
node_video_attention -->|"uses layers"| node_attention
node_spacetime_attention -->|"uses layers"| node_attention
node_conditioning -->|"uses regularizers"| node_regularizers
node_conditioning -->|"uses distributions"| node_distributions
node_regularizers -->|"uses distributions"| node_distributions

click node_sv3d "https://github.com/stability-ai/generative-models/blob/main/scripts/sampling/simple_video_sample.py"
click node_sv4d "https://github.com/stability-ai/generative-models/blob/main/scripts/sampling/simple_video_sample_4d.py"
click node_sv4d2 "https://github.com/stability-ai/generative-models/blob/main/scripts/sampling/simple_video_sample_4d2.py"
click node_inference_api "https://github.com/stability-ai/generative-models/blob/main/sgm/inference/api.py"
click node_inference_helpers "https://github.com/stability-ai/generative-models/blob/main/sgm/inference/helpers.py"
click node_safety_filter "https://github.com/stability-ai/generative-models/blob/main/scripts/util/detection/nsfw_and_watermark_dectection.py"
click node_diffusion_engine "https://github.com/stability-ai/generative-models/blob/main/sgm/models/diffusion.py"
click node_autoencoder "https://github.com/stability-ai/generative-models/blob/main/sgm/models/autoencoder.py"
click node_sampler "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/sampling.py"
click node_conditioning "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/encoders/modules.py"
click node_image_unet "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/openaimodel.py"
click node_video_unet "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/video_model.py"
click node_image_network "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/model.py"
click node_temporal_ae "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/autoencoding/temporal_ae.py"
click node_attention "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/attention.py"
click node_video_attention "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/video_attention.py"
click node_spacetime_attention "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/spacetime_attention.py"
click node_regularizers "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/autoencoding/regularizers/base.py"
click node_distributions "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/distributions/distributions.py"
click node_ema "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/ema.py"
click node_lpips "https://github.com/stability-ai/generative-models/blob/main/sgm/modules/autoencoding/lpips/loss/lpips.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sv3d,node_sv4d,node_sv4d2,node_inference_api,node_inference_helpers,node_safety_filter,node_user toneBlue
class node_diffusion_engine,node_autoencoder,node_sampler,node_conditioning toneAmber
class node_image_unet,node_video_unet,node_image_network,node_temporal_ae,node_attention,node_video_attention,node_spacetime_attention toneMint
class node_regularizers,node_distributions,node_ema,node_lpips toneRose
```

## Components

### Sampling workflows

- [SV3D sampling](https://github.com/stability-ai/generative-models/blob/main/scripts/sampling/simple_video_sample.py): `scripts/sampling/simple_video_sample.py`
- [SV4D sampling](https://github.com/stability-ai/generative-models/blob/main/scripts/sampling/simple_video_sample_4d.py): `scripts/sampling/simple_video_sample_4d.py`
- [SV4D 2.0 sampling](https://github.com/stability-ai/generative-models/blob/main/scripts/sampling/simple_video_sample_4d2.py): `scripts/sampling/simple_video_sample_4d2.py`
- [Inference API](https://github.com/stability-ai/generative-models/blob/main/sgm/inference/api.py): `sgm/inference/api.py`
- [Inference helpers](https://github.com/stability-ai/generative-models/blob/main/sgm/inference/helpers.py): `sgm/inference/helpers.py`
- [Safety detection](https://github.com/stability-ai/generative-models/blob/main/scripts/util/detection/nsfw_and_watermark_dectection.py): `scripts/util/detection/nsfw_and_watermark_dectection.py`

### Diffusion engine

- [Diffusion engine](https://github.com/stability-ai/generative-models/blob/main/sgm/models/diffusion.py): `sgm/models/diffusion.py`
- [Autoencoder engine](https://github.com/stability-ai/generative-models/blob/main/sgm/models/autoencoder.py): `sgm/models/autoencoder.py`
- [Diffusion sampler](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/sampling.py): `sgm/modules/diffusionmodules/sampling.py`
- [Conditioning encoders](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/encoders/modules.py): `sgm/modules/encoders/modules.py`

### Model networks

- [Image UNet](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/openaimodel.py): `sgm/modules/diffusionmodules/openaimodel.py`
- [Video UNet](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/video_model.py): `sgm/modules/diffusionmodules/video_model.py`
- [Image autoencoder network](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/diffusionmodules/model.py): `sgm/modules/diffusionmodules/model.py`
- [Temporal autoencoding](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/autoencoding/temporal_ae.py): `sgm/modules/autoencoding/temporal_ae.py`
- [Attention layers](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/attention.py): `sgm/modules/attention.py`
- [Video attention](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/video_attention.py): `sgm/modules/video_attention.py`
- [Space-time attention](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/spacetime_attention.py): `sgm/modules/spacetime_attention.py`

### Model support

- [Latent regularizers](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/autoencoding/regularizers/base.py): `sgm/modules/autoencoding/regularizers/base.py`
- [Latent distributions](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/distributions/distributions.py): `sgm/modules/distributions/distributions.py`
- [EMA weights](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/ema.py): `sgm/modules/ema.py`
- [Perceptual metric](https://github.com/stability-ai/generative-models/blob/main/sgm/modules/autoencoding/lpips/loss/lpips.py): `sgm/modules/autoencoding/lpips/loss/lpips.py`

### Other components

- **Research user**

## Connections

- Research user → SV3D sampling: supplies image
- Research user → SV4D sampling: supplies video
- Research user → SV4D 2.0 sampling: supplies video
- SV3D sampling → Inference helpers: uses helpers (evidence: `scripts/sampling/simple_video_sample.py`)
- SV3D sampling → Safety detection: imports detector (evidence: `scripts/sampling/simple_video_sample.py`)
- SV3D sampling → Diffusion engine: runs inference
- SV4D sampling → Diffusion engine: runs inference
- SV4D 2.0 sampling → Diffusion engine: runs inference
- SV4D sampling → SV3D sampling: generates reference views
- Inference API → Inference helpers: uses helpers (evidence: `sgm/inference/api.py`)
- Inference API → Diffusion sampler: calls sampler (evidence: `sgm/inference/api.py`)
- Inference helpers → Diffusion engine: denoises and decodes (evidence: `sgm/inference/helpers.py`)
- Diffusion engine → Temporal autoencoding: configures temporal stage (evidence: `sgm/models/diffusion.py`)
- Diffusion engine → EMA weights: tracks model weights (evidence: `sgm/models/diffusion.py`)
- Autoencoder engine → Latent regularizers: configures regularizer (evidence: `sgm/models/autoencoder.py`)
- Autoencoder engine → EMA weights: uses EMA (evidence: `sgm/models/autoencoder.py`)
- Temporal autoencoding → Image autoencoder network: uses network blocks (evidence: `sgm/modules/autoencoding/temporal_ae.py`)
- Temporal autoencoding → Image UNet: uses network blocks (evidence: `sgm/modules/autoencoding/temporal_ae.py`)
- Temporal autoencoding → Video attention: uses attention (evidence: `sgm/modules/autoencoding/temporal_ae.py`)
- Video UNet → Image UNet: extends UNet blocks (evidence: `sgm/modules/diffusionmodules/video_model.py`)
- Video UNet → Video attention: uses attention (evidence: `sgm/modules/diffusionmodules/video_model.py`)
- Video UNet → Space-time attention: uses attention (evidence: `sgm/modules/diffusionmodules/video_model.py`)
- Image UNet → Attention layers: uses attention (evidence: `sgm/modules/diffusionmodules/openaimodel.py`)
- Video attention → Attention layers: uses layers (evidence: `sgm/modules/video_attention.py`)
- Space-time attention → Attention layers: uses layers (evidence: `sgm/modules/spacetime_attention.py`)
- Conditioning encoders → Latent regularizers: uses regularizers (evidence: `sgm/modules/encoders/modules.py`)
- Conditioning encoders → Latent distributions: uses distributions (evidence: `sgm/modules/encoders/modules.py`)
- Latent regularizers → Latent distributions: uses distributions (evidence: `sgm/modules/autoencoding/regularizers/__init__.py`)
