How yang-song/score_sde_pytorch works
Project Type and Purpose
• This is a research‐oriented, full-stack deep-learning codebase for training and evaluating score-based generative models formulated as Stochastic Differential Equations (SDEs).
• It implements multiple model families (NCSN, NCSNv2, NCSN++, DDPM, DDPM++) in PyTorch, provides a unified API for training, sampling, likelihood evaluation, and supports extensibility for new SDEs, predictors, and correctors.
High-Level File/Module Structure
• configs/ – ML-Collections config files that define SDE variants, datasets, model architectures, and training/eval parameters
• models/ – Neural network definitions and building blocks (layers, normalization, EMA, up/down sampling)
• op/ – Custom C++/CUDA ops for high-performance components (fused activations, upfirdn2d)
• sde_lib.py – Abstract SDE class and implementations (VE, VP, subVP)
• sampling.py – Predictor-Corrector and ODE samplers, registry of predictor/corrector classes
• controllable_generation.py – Conditional samplers (inpainting, class-conditional, colorization)
• datasets.py – Data loading and preprocessing pipelines for CIFAR-10, CelebA, LSUN, etc.
• losses.py – Score matching and denoising loss functions
• likelihood.py – Log-likelihood computation via probability flow ODE
• evaluation.py – Metric computation (FID, IS, KID) and sample/likelihood evaluation orchestration
• main.py & run_lib.py – CLI entrypoints for train/eval modes, argument parsing, experiment management (checkpoints, workdir)
• utils.py & debug.py – Utility functions and debugging helpers
Core Architectural Components
A. Configuration & Initialization
– ml_collections drives dynamic instantiation of SDEs, models, datasets, optimizers, samplers
– main.py parses CLI args (mode, config path, workdir), then delegates to run_lib
B. Data Pipeline
– datasets.py defines DataLoader assemblies per dataset config
– assets/stats/ holds precomputed statistics for FID evaluation
C. Model & SDE Definition
– models/ contains network architectures registered by name in configs
– sde_lib.SDE abstract class captures forward (perturb) and reverse (sampling) dynamics
D. Training Loop
– run_lib orchestrates:
• Sample minibatch from data loader
• Forward through SDE perturbation, compute model score via models/*
• Compute loss via losses.py
• Backprop & optimizer step
• Periodic checkpointing (including meta-checkpoints for pre-emption)
E. Sampling & Evaluation
– sampling.get_pc_sampler constructs Predictor-Corrector or ODE samplers using registered predictors/correctors
– controllable_generation.py extends sampling for conditional tasks
– evaluation.py calls sampling pipeline, computes FID/IS/KID, and logs results
– likelihood.py uses the probability flow ODE to compute bits/dim
F. Custom Ops
– op/ enhances performance for critical kernels (activations, downsampling) compiled at install time
Relationships and Data Flow
1. User invokes main.py with CLI → loads config
2. Config → instantiate SDE, model, dataset, optimizer, sampler
3. Training Mode: data → model + sde_lib → losses → optimizer → checkpoints
4. Eval Mode: load checkpoint → sampling.py / controllable_generation → generated samples → evaluation.py → metrics
5. External Resources:
• assets/stats/ (downloaded CIFAR-10 stats for FID)
• Google Drive for pretrained checkpoints
• Optional integration with 🤗 Diffusers for plug-and-play inference
Architectural Patterns & Principles
• Modular Plugin Architecture: new SDEs, predictors, correctors, or model blocks via abstract base classes and registries
• Configuration-Driven Dependency Injection: behavior and components driven entirely by ml_collections configs
• Separation of Concerns: clear demarcation between data, model definition, training, sampling, evaluation
• Performance-Critical Path Optimization: custom ops in C++/CUDA where needed