How crowsonkb/k-diffusion works
Project type and purpose
• k-diffusion is primarily an open-source machine-learning library and command-line tool for training and sampling diffusion-based generative models in PyTorch.
• It includes:
• Core library code under k_diffusion/
• Training (train.py) and sampling (sample.py, sample_clip_guided.py) scripts
• Configuration files to customize datasets, model architectures, and training parameters
File structure
• Top-level scripts:
• train.py – entry point for training runs
• sample.py, sample_clip_guided.py – entry points for inference/sampling
• convert_for_inference.py, config_from_inference.py, make_grid.py – utilities
• Configuration folder: configs/ stores JSON specs for datasets (MNIST, CIFAR-10, Oxford Flowers) and model types (transformer vs. hourglass)
• Core package: k_diffusion/ subdivided into:
• augmentation.py, evaluation.py, sampling.py – pipelines for data and sampling
• models/ – different diffusion and transformer architectures
• layers, utils, external wrappers, config, gns modules
• Build & packaging: setup.py, pyproject.toml, requirements.txt
Extract README insights
• Project uses PyTorch, Hugging Face Datasets, Accelerate for multi-GPU training
• Optional dependencies: FlashAttention-2 and NATTEN for custom CUDA kernels
• Supports CLIP-guided sampling, FID/KID and gradient-noise metrics
• Wraps external diffusion implementations (OpenAI, CompVis, v-diffusion-pytorch)
Main components and interactions
a. Components
1. CLI layer
– train.py
– sample.py / sample_clip_guided.py
2. Configuration loader
– parses JSON from configs/
3. Dataset provider
– built-in (MNIST, CIFAR-10, ImageFolder)
– Hugging Face Datasets
4. Core k-diffusion library
– augmentation
– model definitions (k_diffusion.models.*)
– sampling algorithms (ODE/SDE, DPM-Solver, Karras samplers)
– loss & evaluation (Min-SNR, FID/KID, log-likelihood)
– external wrappers for other diffusion libraries
5. Training orchestrator
– Hugging Face Accelerate or plain PyTorch
– checkpointing, mixed precision, gradient-noise scale computation
6. Inference pipeline
– model loading, sampling loops, optional CLIP guidance
7. External dependencies/services
– CUDA-kernels (FlashAttention-2, NATTEN)
– File system or database for datasets and checkpoints
8. Visualization & utilities
– make_grid.py (image grid assembly)
– config_from_inference / convert_for_inference
b. Relationships & data-flows
– CLI → Configuration loader → Dataset provider + Core library → Training orchestrator → Checkpoint store + Metrics
– CLI (sample) → Configuration loader → Core library (sampling) → Output image grid
– Core library uses external CUDA modules if available, else falls back to PyTorch
– Evaluation modules read training outputs to compute FID/KID
c. Architectural patterns & principles
– Modular design: separation of CLI, config, data, models, sampling, evaluation
– Plug-in/extensible: wrappers for third-party diffusion implementations
– Hierarchical model definitions follow an hourglass/transformer pattern
– Configuration-driven: JSON defines dataset, model, training details
d. Key technologies & libraries
– PyTorch (+ torch.compile fallback)
– Hugging Face Datasets & Accelerate
– FlashAttention-2, NATTEN (optional)
– JSON for configuration
– External diffusion libs (v-diffusion-pytorch, guided-diffusion, latent-diffusion)