How compvis/stable-diffusion works
Stable Diffusion is a latent text-to-image diffusion model, with reference workflows for text-to-image, image-to-image, and inpainting. The main sampling path takes a prompt through text conditioning, latent denoising, and autoencoder decoding to rendered images; training and dataset preparation form a separate branch. The README documents safety checking and invisible watermarking in the reference sampling script. Some implementation details were not sampled, so those documented integrations are shown separately; other edges are limited to explicit source references and documented behavior.
Main entry points: txt2img.py Text sampling · ddpm.py Latent diffusion · ddim.py DDIM sampler · main.py Training orchestration
Components 22
User workflows
- Text sampling
scripts/txt2img.py - Image-to-image
scripts/img2img.py - Inpainting
scripts/inpaint.py - KNN image search
scripts/knn2img.py - Safety checker
- Invisible watermark
- Rendered images
scripts/txt2img.py
Latent diffusion model
- Latent diffusion
ldm/models/diffusion/ddpm.py - Text encoder
ldm/modules/encoders/modules.py - Diffusion UNet
ldm/modules/diffusionmodules/openaimodel.py - Attention layers
ldm/modules/attention.py - Transformer layers
ldm/modules/x_transformer.py - Latent autoencoder
ldm/models/autoencoder.py - Gaussian latents
ldm/modules/distributions/distributions.py
Sampling methods
- DDIM sampler
ldm/models/diffusion/ddim.py - PLMS sampler
ldm/models/diffusion/plms.py - DPM solver
ldm/models/diffusion/dpm_solver/dpm_solver.py
Training and data
- Training orchestration
main.py - Image datasets
ldm/data - Image degradation
ldm/modules/image_degradation/bsrgan.py - EMA weights
ldm/modules/ema.py
Other components
- Prompt user
Connections 26
- Prompt user → Text sampling: provides prompt
README.md - Text sampling → Text encoder: encodes text
scripts/txt2img.py - Text sampling → DDIM sampler: selects sampler
scripts/txt2img.py - Text sampling → PLMS sampler: selects sampler
scripts/txt2img.py - Text sampling → DPM solver: selects sampler
scripts/txt2img.py - DDIM sampler → Latent diffusion: denoises with
ldm/models/diffusion/ddim.py - PLMS sampler → Latent diffusion: denoises with
- Latent diffusion → Diffusion UNet: runs denoiser
ldm/models/diffusion/ddpm.py - Latent diffusion → Latent autoencoder: uses first stage
ldm/models/diffusion/ddpm.py - Latent autoencoder → Gaussian latents: models latent distribution
ldm/models/autoencoder.py - Text encoder → Transformer layers: uses transformer
ldm/modules/encoders/modules.py - Diffusion UNet → Attention layers: uses attention
ldm/modules/diffusionmodules/openaimodel.py - Text sampling → Latent autoencoder: decodes samples
- Text sampling → Safety checker: checks outputs
README.md - Text sampling → Invisible watermark: watermarks outputs
README.md - Text sampling → Rendered images: writes images
README.md - Prompt user → Image-to-image: supplies image
scripts/img2img.py - Image-to-image → DDIM sampler: samples with
scripts/img2img.py - Prompt user → Inpainting: supplies mask
scripts/inpaint.py - Inpainting → DDIM sampler: samples with
scripts/inpaint.py - KNN image search → Text encoder: uses encoder
scripts/knn2img.py - KNN image search → PLMS sampler: samples with
scripts/knn2img.py - Training orchestration → Image datasets: loads datasets
main.py - Training orchestration → Latent diffusion: trains model
- Training orchestration → EMA weights: tracks weights
- Image datasets → Image degradation: uses degradation