How hkproj/pytorch-stable-diffusion works
Project type and purpose
• This is a research-style PyTorch implementation of the Stable Diffusion image synthesis pipeline.
• It is structured as a library of modules under the sd/ folder, with demonstration notebooks (.ipynb), weight-downloader instructions in the README, and utilities for converting/loading models.
File structure
• Top level:
• configuration files (.gitignore, requirements.txt, license.txt)
• sd/ directory containing all code modules.
• Inside sd/:
• model_loader.py, model_converter.py – weight/tokenizer loading & conversion utilities.
• encoder.py, decoder.py, attention.py, ddpm.py, diffusion.py – core model building blocks (VAE, UNet, attention, noise schedule).
• clip.py – CLIP text encoder wrapper.
• pipeline.py – high-level class orchestrating the end-to-end forward pass.
• Notebooks (add_noise.ipynb, demo.ipynb) illustrate usage.
README for additional insights
• Describes:
• Downstream use: inference with v1.5 weights and tokenizers.
• Ability to swap in fine-tuned checkpoint files.
a. Main components:
• External Data Store (folder with vocab.json, merges.txt, .ckpt files)
• Tokenizer (Hugging Face vocab + merges)
• Text Encoder (clip.py)
• Model Loader & Converter (model_loader.py, model_converter.py)
• U-Net Diffusion Core (diffusion.py, ddpm.py, attention.py)
• VAE Encoder/Decoder (encoder.py, decoder.py)
• Pipeline Facade (pipeline.py)
• User Interface (Jupyter notebooks or CLI)
b. Relationships & data flow:
1. User provides text prompt to Pipeline
2. Pipeline calls Tokenizer → token IDs
3. Pipeline calls Text Encoder → text embeddings
4. Pipeline initializes latents (random noise)
5. Iterative Diffusion Loop:
– UNet + cross-attention consume latents + text embeddings → denoised latents
– Schedule module in ddpm.py updates noise levels
6. Final latents → VAE Decoder → RGB image
7. Pipeline returns image (or saves to disk)
8. Model Loader underpins all model modules by loading weights at startup.
c. Architectural patterns & principles:
• Modular design: separates concerns (tokenization, text encoding, diffusion, decoding, orchestration).
• Pipeline pattern: a single facade orchestrates multiple steps.
• Lazy-loading and weight conversion utilities abstract away file I/O.
d. Key technologies & frameworks:
• PyTorch (tensor operations, model definition)
• Hugging Face tokenizers (vocab.json, merges.txt)
• Diffusion scheduler algorithms (DDPM style)
• Jupyter notebooks for demo and interactive exploration
Detail without over-splitting
• Weight loading is shown once at startup (Model Loader → all model components).