# crowsonkb/k-diffusion architecture

> Architecture diagram of the GitHub repository crowsonkb/k-diffusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crowsonkb/k-diffusion
- Repository: https://github.com/crowsonkb/k-diffusion

## Overview

##### Project type and purpose
- k-diffusion is primarily an open-source machine-learning library and command-line tool for training and sampling diffusion-based generative models in PyTorch.
- It includes:
- Core library code under `k_diffusion/`
- Training (train.py) and sampling (sample.py, sample_clip_guided.py) scripts
- Configuration files to customize datasets, model architectures, and training parameters

##### File structure
- Top-level scripts:
- train.py – entry point for training runs
- sample.py, sample_clip_guided.py – entry points for inference/sampling
- convert_for_inference.py, config_from_inference.py, make_grid.py – utilities
- Configuration folder: `configs/` stores JSON specs for datasets (MNIST, CIFAR-10, Oxford Flowers) and model types (transformer vs. hourglass)
- Core package: `k_diffusion/` subdivided into:
- augmentation.py, evaluation.py, sampling.py – pipelines for data and sampling
- models/ – different diffusion and transformer architectures
- layers, utils, external wrappers, config, gns modules
- Build & packaging: `setup.py`, `pyproject.toml`, `requirements.txt`

##### Extract README insights
- Project uses PyTorch, Hugging Face Datasets, Accelerate for multi-GPU training
- Optional dependencies: FlashAttention-2 and NATTEN for custom CUDA kernels
- Supports CLIP-guided sampling, FID/KID and gradient-noise metrics
- Wraps external diffusion implementations (OpenAI, CompVis, v-diffusion-pytorch)

##### Main components and interactions
a. Components
1. CLI layer
– train.py
– sample.py / sample_clip_guided.py
2. Configuration loader
– parses JSON from `configs/`
3. Dataset provider
– built-in (MNIST, CIFAR-10, ImageFolder)
– Hugging Face Datasets
4. Core k-diffusion library
– augmentation
– model definitions (k_diffusion.models.*)
– sampling algorithms (ODE/SDE, DPM-Solver, Karras samplers)
– loss & evaluation (Min-SNR, FID/KID, log-likelihood)
– external wrappers for other diffusion libraries
5. Training orchestrator
– Hugging Face Accelerate or plain PyTorch
– checkpointing, mixed precision, gradient-noise scale computation
6. Inference pipeline
– model loading, sampling loops, optional CLIP guidance
7. External dependencies/services
– CUDA-kernels (FlashAttention-2, NATTEN)
– File system or database for datasets and checkpoints
8. Visualization & utilities
– make_grid.py (image grid assembly)
– config_from_inference / convert_for_inference

b. Relationships & data-flows
– CLI → Configuration loader → Dataset provider + Core library → Training orchestrator → Checkpoint store + Metrics
– CLI (sample) → Configuration loader → Core library (sampling) → Output image grid
– Core library uses external CUDA modules if available, else falls back to PyTorch
– Evaluation modules read training outputs to compute FID/KID

c. Architectural patterns & principles
– Modular design: separation of CLI, config, data, models, sampling, evaluation
– Plug-in/extensible: wrappers for third-party diffusion implementations
– Hierarchical model definitions follow an hourglass/transformer pattern
– Configuration-driven: JSON defines dataset, model, training details

d. Key technologies & libraries
– PyTorch (+ torch.compile fallback)
– Hugging Face Datasets & Accelerate
– FlashAttention-2, NATTEN (optional)
– JSON for configuration
– External diffusion libs (v-diffusion-pytorch, guided-diffusion, latent-diffusion)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Classes
    classDef scripts fill:#B3D4FC,stroke:#000,stroke-width:1px
    classDef core fill:#C6F5C6,stroke:#000,stroke-width:1px
    classDef external fill:#FFF2B8,stroke:#000,stroke-width:1px,stroke-dasharray: 5 5
    classDef storage fill:#FFD5A6,stroke:#000,stroke-width:1px

    %% CLI Layer
    subgraph "CLI Layer"
        Train["train.py (CLI)"]:::scripts
        Sample["sample.py (CLI)"]:::scripts
        SampleCLIP["sample_clip_guided.py (CLI)"]:::scripts
    end

    %% Configuration
    ConfigLoader["Config Loader\nk_diffusion/config.py"]:::core
    ConfigFiles["Configuration Files\nconfigs/*.json"]:::storage

    %% Dataset Provider
    DatasetProvider["Dataset Provider\n(JSON → dataset)"]:::core

    %% Core Library
    subgraph "Core Library"
        Aug["augmentation.py"]:::core
        Eval["evaluation.py"]:::core
        Samp["sampling.py"]:::core
        Models["models/"]:::core
        Layers["layers.py"]:::core
        Utils["utils.py"]:::core
        External["external.py"]:::core
        GNS["gns.py"]:::core
    end

    %% Training Orchestrator
    TrainOrch["Training Orchestrator\n(HF Accelerate / PyTorch)"]:::core

    %% Inference Pipeline
    Inference["Inference Pipeline\n(model loading → sampling)"]:::core

    %% Visualization & Utilities
    MakeGrid["make_grid.py"]:::scripts
    ConvertInf["convert_for_inference.py"]:::scripts
    ConfigInf["config_from_inference.py"]:::scripts

    %% Storage
    Checkpoints["Checkpoints"]:::storage
    Datasets["Datasets"]:::storage

    %% External Dependencies
    Flash["FlashAttention-2"]:::external
    NATTEN["NATTEN"]:::external
    ThirdParty["Third-Party Diffusion Libs\n(v-diffusion, CompVis, etc.)"]:::external

    %% Connections - Training Flow
    Train -->|reads| ConfigLoader
    ConfigLoader -->|loads JSON| ConfigFiles
    ConfigLoader -->|provides settings| DatasetProvider
    DatasetProvider -->|loads data| Datasets
    ConfigLoader -->|provides settings| TrainOrch
    TrainOrch -->|calls| Aug
    TrainOrch -->|instantiates| Models
    TrainOrch -->|uses| Layers
    TrainOrch -->|uses| Utils
    TrainOrch -->|computes| GNS
    TrainOrch -->|evaluates| Eval
    TrainOrch -->|samples| Samp
    TrainOrch -->|wraps| External
    TrainOrch -->|writes| Checkpoints

    %% Connections - Sampling Flow
    Sample -->|reads| ConfigLoader
    SampleCLIP -->|reads| ConfigLoader
    ConfigLoader -->|provides settings| Inference
    Inference -->|uses| Models
    Inference -->|uses| Layers
    Inference -->|uses| Samp
    Inference -->|wraps| External
    Inference -->|outputs| MakeGrid

    %% Utilities
    Inference -->|to| ConvertInf
    Inference -->|to| ConfigInf
    MakeGrid -->|reads| Checkpoints

    %% External Dependencies Links
    Aug -.-> Flash
    Models -.-> Flash
    Samp -.-> NATTEN
    External --> ThirdParty

    %% Click Events
    click Train "https://github.com/crowsonkb/k-diffusion/blob/master/train.py"
    click Sample "https://github.com/crowsonkb/k-diffusion/blob/master/sample.py"
    click SampleCLIP "https://github.com/crowsonkb/k-diffusion/blob/master/sample_clip_guided.py"
    click ConfigLoader "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/config.py"
    click ConfigFiles "https://github.com/crowsonkb/k-diffusion/tree/master/configs/"
    click Aug "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/augmentation.py"
    click Eval "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/evaluation.py"
    click Samp "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/sampling.py"
    click Models "https://github.com/crowsonkb/k-diffusion/tree/master/k_diffusion/models/"
    click Layers "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/layers.py"
    click Utils "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/utils.py"
    click External "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/external.py"
    click GNS "https://github.com/crowsonkb/k-diffusion/blob/master/k_diffusion/gns.py"
    click MakeGrid "https://github.com/crowsonkb/k-diffusion/blob/master/make_grid.py"
    click ConvertInf "https://github.com/crowsonkb/k-diffusion/blob/master/convert_for_inference.py"
    click ConfigInf "https://github.com/crowsonkb/k-diffusion/blob/master/config_from_inference.py"
```
