How openai/improved-diffusion works
1. This repository is a research training/sampling package for diffusion models, centered on the improved_diffusion Python package and a small set of CLI scripts. It is not a web service; the main runtime is local or distributed Python jobs launched from the command line.
2. The top-level architecture splits into four parts: model/diffusion core (improved_diffusion/), dataset preparation helpers (datasets/), executable entrypoints (scripts/), and packaging metadata (setup.py). The README describes the intended flow: prepare image folders, train a model, then sample from saved checkpoints.
3. The core model subsystem is built around a U-Net implementation in improved_diffusion/unet.py. This is the neural network that predicts denoising outputs during training and generation, with configuration assembled through script_util.py.
4. The diffusion-process subsystem lives in improved_diffusion/gaussian_diffusion.py, with timestep/respacing logic in respace.py and resample.py. This layer defines the forward noising process, reverse sampling process, loss targets, and optional timestep selection strategies such as reduced-step sampling and DDIM-style spacing.
5. Training orchestration is handled by improved_diffusion/train_util.py. It connects the data loader, model, diffusion object, optimizer/training loop, checkpointing, and logging. The losses.py module holds the objective calculations used during optimization, including variants such as standard diffusion loss and VLB-related terms.
6. Data ingestion is intentionally simple and file-system based. improved_diffusion/image_datasets.py loads images from a directory tree, handles resizing/cropping, and optionally infers class labels from filename prefixes or subdirectories. The datasets/ folder contains dataset-specific preparation scripts for CIFAR-10 and LSUN bedroom, plus README instructions for building these directories.
7. The executable scripts provide the user-facing boundaries of the system. scripts/image_train.py trains unconditional or class-conditional image models, scripts/image_sample.py loads a checkpoint and produces samples, scripts/image_nll.py evaluates likelihood-related metrics, and scripts/super_res_train.py / scripts/super_res_sample.py support conditional super-resolution workflows.
8. The super-resolution path is a distinct conditional pipeline rather than just another sampling mode. It reuses the same diffusion core and model machinery, but the inputs include a low-resolution conditioning image, so the model boundary differs from pure unconditional generation.
9. Distributed and memory-efficiency concerns are handled explicitly inside the package. dist_util.py supports MPI-style multi-GPU/multi-machine execution, while fp16_util.py provides mixed-precision helpers. The README’s mpiexec examples and --microbatch flag reflect this architecture: the training loop is designed to scale across ranks but still work on a single GPU.
10. Logging and artifact flow are file-based. logger.py writes metrics, checkpoints, and sample outputs to a run directory controlled by OPENAI_LOGDIR or a temporary directory. Training produces .pt checkpoints; sampling writes .npz batches, with the EMA checkpoint typically serving as the preferred sampling source.
11. script_util.py is the configuration boundary that converts CLI flags into model, diffusion, and training objects. This is important for diagramming because the scripts do not hardcode architectures; instead, they assemble experiments from a shared parameterization of image size, channel count, residual blocks, diffusion steps, noise schedule, and conditioning mode.
12. Overall data flow is: image files on disk → dataset loader/preprocessing → U-Net + Gaussian diffusion training loop → checkpoint files and logs → sampling/evaluation scripts that reload checkpoints and run the reverse diffusion process to emit images or likelihood outputs.
13. The repo is self-contained aside from standard Python dependencies and MPI support. There are no external APIs or databases; the only notable external interaction is the filesystem for datasets, checkpoints, logs, and generated sample files.
Main entry points: gaussian_diffusion.py Diffusion process · train_util.py Train loop · script_util.py Config builder · logger.py Run logger
Components 20
Diffusion core
Model, diffusion, training, and runtime utilities inside the Python package.
- Config builder · Converts CLI flags into model, diffusion, and training objects.
improved_diffusion/script_util.py - U-Net · Neural network backbone used to predict denoising outputs.
improved_diffusion/unet.py - Diffusion process · Forward noising, reverse sampling, and diffusion loss targets.
improved_diffusion/gaussian_diffusion.py - Timestep spacing · Reduces or rearranges timesteps for faster or alternative sampling.
improved_diffusion/respace.py - Timestep sampler · Chooses timestep selection strategies used during training or sampling.
improved_diffusion/resample.py - Train loop · Connects data, model, optimizer, checkpoints, and logging.
improved_diffusion/train_util.py - Objectives · Computes diffusion training objectives and VLB-related terms.
improved_diffusion/losses.py - Distributed I/O · MPI-style multi-GPU and multi-machine helpers.
improved_diffusion/dist_util.py - FP16 helpers · Mixed-precision training and model utilities.
improved_diffusion/fp16_util.py - Run logger · Writes metrics, checkpoints, and sample outputs to the run directory.
improved_diffusion/logger.py - NN utilities · Shared neural-network helper functions used by core modules.
improved_diffusion/nn.py
Data prep
Filesystem-based dataset loading and dataset-specific preparation helpers.
- Image loader · Loads images from directories, preprocesses them, and infers labels when available.
improved_diffusion/image_datasets.py - CIFAR-10 prep · Prepares CIFAR-10 data into the expected filesystem layout.
datasets/cifar10.py - LSUN prep · Prepares LSUN bedroom data into the expected filesystem layout.
datasets/lsun_bedroom.py
Entry scripts
Command-line entrypoints for training, sampling, likelihood evaluation, and super-resolution.
- Train images · Main entrypoint for unconditional and class-conditional training.
scripts/image_train.py - Sample images · Loads a checkpoint and generates image samples.
scripts/image_sample.py - Evaluate NLL · Computes likelihood-related metrics from a trained checkpoint.
scripts/image_nll.py - Train SR · Trains the conditional super-resolution model path.
scripts/super_res_train.py - Sample SR · Runs super-resolution sampling from a checkpoint and low-resolution input.
scripts/super_res_sample.py
Other components
- Package metadata · Python packaging metadata for installing the repository.
setup.py
Connections 30
- Train images → Config builder: builds · Assembles the experiment configuration from CLI flags.
- Sample images → Config builder: builds · Uses the shared config boundary to construct sampling components.
- Evaluate NLL → Config builder: builds · Constructs evaluation-time model and diffusion objects.
- Train SR → Config builder: builds · Creates the conditional super-resolution training setup.
- Sample SR → Config builder: builds · Creates the conditional super-resolution sampling setup.
- Config builder → U-Net: configures · Selects the U-Net architecture and its conditioning parameters.
- Config builder → Diffusion process: configures · Creates the diffusion process from schedule and loss settings.
- Train loop → Image loader: loads data · Consumes image batches from the filesystem dataset loader.
- Train loop → U-Net: trains · Optimizes the denoising network during training.
- Train loop → Diffusion process: optimizes · Uses diffusion objectives to compute training signals.
- Train loop → Objectives: computes · Evaluates diffusion and VLB-style objectives.
- Train loop → Distributed I/O: runs on · Supports distributed execution across ranks and devices.
- Train loop → FP16 helpers: uses · Applies mixed-precision helpers when enabled.
- Train loop → Run logger: writes · Persists checkpoints, metrics, and sample artifacts.
- Diffusion process → Timestep spacing: uses · Applies timestep spacing for reduced-step sampling.
- Diffusion process → Timestep sampler: uses · Selects timestep schedules and resampling strategies.
- Sample images → Diffusion process: samples · Runs the reverse diffusion process to generate images.
- Evaluate NLL → Diffusion process: evaluates · Uses diffusion math to compute likelihood-related outputs.
- Sample SR → Diffusion process: samples · Uses the reverse process with low-resolution conditioning.
- Train SR → Diffusion process: trains · Optimizes the conditional diffusion objective for super-resolution.
- Image loader → CIFAR-10 prep: expects · Consumes CIFAR-10 directories prepared by dataset scripts.
- Image loader → LSUN prep: expects · Consumes LSUN bedroom directories prepared by dataset scripts.
- CIFAR-10 prep → Package metadata: packages with · Dataset tooling lives alongside the installable Python package.
- LSUN prep → Package metadata: packages with · Dataset tooling lives alongside the installable Python package.
- Train images → Run logger: writes · Training entrypoint writes run artifacts through the logger.
- Sample images → Run logger: writes · Sampling entrypoint emits generated samples to the run directory.
- Sample SR → Run logger: writes · Super-resolution sampling writes outputs and metadata.
- U-Net → NN utilities: builds on · Uses shared neural-network helper functions.
- Diffusion process → NN utilities: builds on · Relies on shared tensor and module utilities.
- Package metadata → Distributed I/O: installs · Packaging exposes the distributed runtime helpers as part of the library.