# yang-song/score_sde_pytorch architecture

> Architecture diagram of the GitHub repository yang-song/score_sde_pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yang-song/score_sde_pytorch
- Repository: https://github.com/yang-song/score_sde_pytorch

## Overview

##### Project Type and Purpose
- This is a research‐oriented, full-stack deep-learning codebase for training and evaluating score-based generative models formulated as Stochastic Differential Equations (SDEs).
- It implements multiple model families (NCSN, NCSNv2, NCSN++, DDPM, DDPM++) in PyTorch, provides a unified API for training, sampling, likelihood evaluation, and supports extensibility for new SDEs, predictors, and correctors.

##### High-Level File/Module Structure
- configs/ – ML-Collections config files that define SDE variants, datasets, model architectures, and training/eval parameters
- models/ – Neural network definitions and building blocks (layers, normalization, EMA, up/down sampling)
- op/ – Custom C++/CUDA ops for high-performance components (fused activations, upfirdn2d)
- sde_lib.py – Abstract SDE class and implementations (VE, VP, subVP)
- sampling.py – Predictor-Corrector and ODE samplers, registry of predictor/corrector classes
- controllable_generation.py – Conditional samplers (inpainting, class-conditional, colorization)
- datasets.py – Data loading and preprocessing pipelines for CIFAR-10, CelebA, LSUN, etc.
- losses.py – Score matching and denoising loss functions
- likelihood.py – Log-likelihood computation via probability flow ODE
- evaluation.py – Metric computation (FID, IS, KID) and sample/likelihood evaluation orchestration
- main.py & run_lib.py – CLI entrypoints for train/eval modes, argument parsing, experiment management (checkpoints, workdir)
- utils.py & debug.py – Utility functions and debugging helpers

##### Core Architectural Components
A. Configuration & Initialization
– ml_collections drives dynamic instantiation of SDEs, models, datasets, optimizers, samplers
– main.py parses CLI args (mode, config path, workdir), then delegates to run_lib
B. Data Pipeline
– datasets.py defines DataLoader assemblies per dataset config
– assets/stats/ holds precomputed statistics for FID evaluation
C. Model & SDE Definition
– models/ contains network architectures registered by name in configs
– sde_lib.SDE abstract class captures forward (perturb) and reverse (sampling) dynamics
D. Training Loop
– run_lib orchestrates:
- Sample minibatch from data loader
- Forward through SDE perturbation, compute model score via models/*
- Compute loss via losses.py
- Backprop & optimizer step
- Periodic checkpointing (including meta-checkpoints for pre-emption)
E. Sampling & Evaluation
– sampling.get_pc_sampler constructs Predictor-Corrector or ODE samplers using registered predictors/correctors
– controllable_generation.py extends sampling for conditional tasks
– evaluation.py calls sampling pipeline, computes FID/IS/KID, and logs results
– likelihood.py uses the probability flow ODE to compute bits/dim
F. Custom Ops
– op/ enhances performance for critical kernels (activations, downsampling) compiled at install time

##### Relationships and Data Flow
1. User invokes main.py with CLI → loads config
2. Config → instantiate SDE, model, dataset, optimizer, sampler
3. Training Mode: data → model + sde_lib → losses → optimizer → checkpoints
4. Eval Mode: load checkpoint → sampling.py / controllable_generation → generated samples → evaluation.py → metrics
5. External Resources:
- assets/stats/ (downloaded CIFAR-10 stats for FID)
- Google Drive for pretrained checkpoints
- Optional integration with 🤗 Diffusers for plug-and-play inference

##### Architectural Patterns & Principles
- Modular Plugin Architecture: new SDEs, predictors, correctors, or model blocks via abstract base classes and registries
- Configuration-Driven Dependency Injection: behavior and components driven entirely by ml_collections configs
- Separation of Concerns: clear demarcation between data, model definition, training, sampling, evaluation
- Performance-Critical Path Optimization: custom ops in C++/CUDA where needed

## Diagram (Mermaid)

```mermaid
graph TB
    %% Subgraphs
    subgraph "Configuration Layer"
        direction TB
        CFG_CONFIGS["configs/"]:::orchestration
        CFG_MAIN["main.py"]:::orchestration
        CFG_RUNLIB["run_lib.py"]:::orchestration
    end

    subgraph "Data Layer"
        direction TB
        DATASETS["datasets.py"]:::data
        ASSETS["assets/"]:::data
    end

    subgraph "Model Layer"
        direction TB
        MODELS_DIR["models/"]:::core
        DDPM["ddpm.py"]:::core
        NCSNV2["ncsnv2.py"]:::core
        NCSNPP["ncsnpp.py"]:::core
        LAYERS["layers.py"]:::core
        LAYERSPP["layerspp.py"]:::core
        NORM["normalization.py"]:::core
        UPSAMP["up_or_down_sampling.py"]:::core
        EMA["ema.py"]:::core
        MODELS_UTIL["utils.py"]:::core
    end

    subgraph "Custom Ops Layer"
        direction TB
        OP_DIR["op/"]:::util
        FUSED_ACT_PY["fused_act.py"]:::util
        FUSED_BIAS_CPP["fused_bias_act.cpp"]:::util
        FUSED_BIAS_CU["fused_bias_act_kernel.cu"]:::util
        UPFIRDN2D_PY["upfirdn2d.py"]:::util
        UPFIRDN2D_CPP["upfirdn2d.cpp"]:::util
        UPFIRDN2D_CU["upfirdn2d_kernel.cu"]:::util
    end

    subgraph "SDE & Sampling Layer"
        direction TB
        SDE_LIB["sde_lib.py"]:::core
        SAMPLING["sampling.py"]:::core
        CONT_GEN["controllable_generation.py"]:::core
    end

    subgraph "Training Engine"
        direction TB
        TRAIN_RUNLIB["run_lib.py"]:::orchestration
        LOSSES["losses.py"]:::orchestration
        OPTIM["optimizers"]:::orchestration
        CHECKPOINTS["Checkpoints"]:::orchestration
    end

    subgraph "Evaluation Engine"
        direction TB
        EVAL["evaluation.py"]:::orchestration
        LIKELIHOOD["likelihood.py"]:::orchestration
        METRICS["Metrics (FID/IS/KID)"]:::orchestration
    end

    subgraph "Utilities"
        direction TB
        UTIL["utils.py"]:::util
        DEBUG["debug.py"]:::util
    end

    subgraph "External Services"
        direction TB
        PT["PyTorch"]:::external
        ML_COL["ml_collections"]:::external
        CUDA["CUDA"]:::external
        GD["Google Drive"]:::external
        DIFF["🤗 Diffusers"]:::external
    end

    %% Connections
    CFG_MAIN -->|"parses CLI"| CFG_RUNLIB
    CFG_CONFIGS -->|"drives instantiation"| CFG_RUNLIB
    CFG_RUNLIB -->|"load data pipeline"| DATASETS
    DATASETS -->|"provides minibatch"| TRAIN_RUNLIB
    CFG_RUNLIB -->|"instantiate model"| MODELS_DIR
    MODELS_DIR --> DDPM
    MODELS_DIR --> NCSNV2
    MODELS_DIR --> NCSNPP
    MODELS_DIR --> LAYERS
    MODELS_DIR --> LAYERSPP
    MODELS_DIR --> NORM
    MODELS_DIR --> UPSAMP
    MODELS_DIR --> EMA
    MODELS_DIR --> MODELS_UTIL
    CFG_RUNLIB -->|"instantiate SDE"| SDE_LIB
    SDE_LIB -->|"perturb/reverse"| TRAIN_RUNLIB
    TRAIN_RUNLIB -->|"forward/backprop"| LOSSES
    LOSSES -->|"compute gradients"| OPTIM
    OPTIM -->|"step"| TRAIN_RUNLIB
    TRAIN_RUNLIB -->|"save"| CHECKPOINTS
    TRAIN_RUNLIB -->|"load checkpoint"| EVAL
    SDE_LIB --> SAMPLING
    SAMPLING --> CONT_GEN
    CONT_GEN --> EVAL
    EVAL --> METRICS
    EVAL -->|"bits/dim via ODE"| LIKELIHOOD
    ASSETS --> EVAL
    OPTIM --> PT
    MODELS_DIR --> PT
    LOSSES --> PT
    SAMPLING --> PT
    SDE_LIB --> PT
    OP_DIR --> FUSED_ACT_PY
    OP_DIR --> FUSED_BIAS_CPP
    OP_DIR --> FUSED_BIAS_CU
    OP_DIR --> UPFIRDN2D_PY
    OP_DIR --> UPFIRDN2D_CPP
    OP_DIR --> UPFIRDN2D_CU
    TRAIN_RUNLIB --> OP_DIR
    MODELS_DIR --> OP_DIR
    PT --> CUDA
    ML_COL --> CFG_CONFIGS
    GD --> CHECKPOINTS
    DIFF --> EVAL

    %% Click Events
    click CFG_CONFIGS "https://github.com/yang-song/score_sde_pytorch/tree/main/./configs/"
    click CFG_MAIN "https://github.com/yang-song/score_sde_pytorch/blob/main/./main.py"
    click CFG_RUNLIB "https://github.com/yang-song/score_sde_pytorch/blob/main/./run_lib.py"
    click DATASETS "https://github.com/yang-song/score_sde_pytorch/blob/main/./datasets.py"
    click ASSETS "https://github.com/yang-song/score_sde_pytorch/tree/main/./assets/"
    click MODELS_DIR "https://github.com/yang-song/score_sde_pytorch/tree/main/./models/"
    click DDPM "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/ddpm.py"
    click NCSNV2 "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/ncsnv2.py"
    click NCSNPP "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/ncsnpp.py"
    click LAYERS "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/layers.py"
    click LAYERSPP "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/layerspp.py"
    click NORM "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/normalization.py"
    click UPSAMP "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/up_or_down_sampling.py"
    click EMA "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/ema.py"
    click MODELS_UTIL "https://github.com/yang-song/score_sde_pytorch/blob/main/./models/utils.py"
    click OP_DIR "https://github.com/yang-song/score_sde_pytorch/tree/main/./op/"
    click FUSED_ACT_PY "https://github.com/yang-song/score_sde_pytorch/blob/main/./op/fused_act.py"
    click FUSED_BIAS_CPP "https://github.com/yang-song/score_sde_pytorch/blob/main/./op/fused_bias_act.cpp"
    click FUSED_BIAS_CU "https://github.com/yang-song/score_sde_pytorch/blob/main/./op/fused_bias_act_kernel.cu"
    click UPFIRDN2D_PY "https://github.com/yang-song/score_sde_pytorch/blob/main/./op/upfirdn2d.py"
    click UPFIRDN2D_CPP "https://github.com/yang-song/score_sde_pytorch/blob/main/./op/upfirdn2d.cpp"
    click UPFIRDN2D_CU "https://github.com/yang-song/score_sde_pytorch/blob/main/./op/upfirdn2d_kernel.cu"
    click SDE_LIB "https://github.com/yang-song/score_sde_pytorch/blob/main/./sde_lib.py"
    click SAMPLING "https://github.com/yang-song/score_sde_pytorch/blob/main/./sampling.py"
    click CONT_GEN "https://github.com/yang-song/score_sde_pytorch/blob/main/./controllable_generation.py"
    click LOSSES "https://github.com/yang-song/score_sde_pytorch/blob/main/./losses.py"
    click EVAL "https://github.com/yang-song/score_sde_pytorch/blob/main/./evaluation.py"
    click LIKELIHOOD "https://github.com/yang-song/score_sde_pytorch/blob/main/./likelihood.py"
    click UTIL "https://github.com/yang-song/score_sde_pytorch/blob/main/./utils.py"
    click DEBUG "https://github.com/yang-song/score_sde_pytorch/blob/main/./debug.py"

    %% Styles
    classDef core fill:#d0e1f9,stroke:#2b6cb0,color:#2b6cb0;
    classDef orchestration fill:#d4edda,stroke:#2f855a,color:#2f855a;
    classDef data fill:#fff3cd,stroke:#dd6b20,color:#dd6b20;
    classDef util fill:#e2e8f0,stroke:#4a5568,color:#4a5568;
    classDef external fill:#ffffff,stroke-dasharray: 2 2,stroke:#718096,color:#718096;
```
