# hkproj/pytorch-stable-diffusion architecture

> Architecture diagram of the GitHub repository hkproj/pytorch-stable-diffusion, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hkproj/pytorch-stable-diffusion
- Repository: https://github.com/hkproj/pytorch-stable-diffusion

## Overview

##### Project type and purpose
- This is a research-style PyTorch implementation of the Stable Diffusion image synthesis pipeline.
- It is structured as a library of modules under the `sd/` folder, with demonstration notebooks (`.ipynb`), weight-downloader instructions in the README, and utilities for converting/loading models.

##### File structure
- Top level:
- configuration files (`.gitignore`, `requirements.txt`, `license.txt`)
- `sd/` directory containing all code modules.
- Inside `sd/`:
- `model_loader.py`, `model_converter.py` – weight/tokenizer loading & conversion utilities.
- `encoder.py`, `decoder.py`, `attention.py`, `ddpm.py`, `diffusion.py` – core model building blocks (VAE, UNet, attention, noise schedule).
- `clip.py` – CLIP text encoder wrapper.
- `pipeline.py` – high-level class orchestrating the end-to-end forward pass.
- Notebooks (`add_noise.ipynb`, `demo.ipynb`) illustrate usage.

##### README for additional insights
- Describes:
- Downstream use: inference with v1.5 weights and tokenizers.
- Ability to swap in fine-tuned checkpoint files.

a. Main components:
- External Data Store (folder with `vocab.json`, `merges.txt`, `.ckpt` files)
- Tokenizer (Hugging Face vocab + merges)
- Text Encoder (`clip.py`)
- Model Loader & Converter (`model_loader.py`, `model_converter.py`)
- U-Net Diffusion Core (`diffusion.py`, `ddpm.py`, `attention.py`)
- VAE Encoder/Decoder (`encoder.py`, `decoder.py`)
- Pipeline Facade (`pipeline.py`)
- User Interface (Jupyter notebooks or CLI)
b. Relationships & data flow:
1. User provides text prompt to Pipeline
2. Pipeline calls Tokenizer → token IDs
3. Pipeline calls Text Encoder → text embeddings
4. Pipeline initializes latents (random noise)
5. Iterative Diffusion Loop:
– UNet + cross-attention consume latents + text embeddings → denoised latents
– Schedule module in `ddpm.py` updates noise levels
6. Final latents → VAE Decoder → RGB image
7. Pipeline returns image (or saves to disk)
8. Model Loader underpins all model modules by loading weights at startup.
c. Architectural patterns & principles:
- Modular design: separates concerns (tokenization, text encoding, diffusion, decoding, orchestration).
- Pipeline pattern: a single facade orchestrates multiple steps.
- Lazy-loading and weight conversion utilities abstract away file I/O.
d. Key technologies & frameworks:
- PyTorch (tensor operations, model definition)
- Hugging Face tokenizers (vocab.json, merges.txt)
- Diffusion scheduler algorithms (DDPM style)
- Jupyter notebooks for demo and interactive exploration

##### Detail without over-splitting
- Weight loading is shown once at startup (Model Loader → all model components).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interface"
        Demo["Demo Notebook"]:::ui
        NoiseNB["Add Noise Notebook"]:::ui
    end
    Demo --> Pipeline
    NoiseNB --> Pipeline

    Pipeline["Pipeline Facade"]:::orchestrator
    click Pipeline "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/pipeline.py"

    subgraph "Model Setup"
        ModelLoader["Model Loader"]:::model
        ModelConverter["Model Converter"]:::model
    end
    Pipeline --> ModelLoader
    Pipeline --> ModelConverter

    ExternalData[(External Data Store)]:::external
    ExternalData --> ModelLoader
    ExternalData --> Tokenizer
    ExternalData --> ModelConverter

    Tokenizer["Tokenizer"]:::model
    Pipeline --> Tokenizer

    TextEncoder["Text Encoder"]:::model
    Pipeline --> TextEncoder
    Tokenizer --> TextEncoder

    ModelLoader --> TextEncoder

    subgraph "Model Components"
        subgraph "Diffusion UNet Core"
            UNet["U-Net"]:::model
            Attention["Attention"]:::model
            Scheduler["Scheduler (DDPM)"]:::model
        end
        subgraph "VAE"
            VAE_Enc["VAE Encoder"]:::model
            VAE_Dec["VAE Decoder"]:::model
        end
    end

    ModelLoader --> UNet
    ModelLoader --> VAE_Enc
    ModelLoader --> VAE_Dec

    TextEncoder --> Embeddings["Text Embeddings"]:::model
    Embeddings --> UNet

    UNet --> Scheduler
    Scheduler --> UNet
    UNet --> Latents["Denoised Latents"]:::model

    Latents --> VAE_Dec
    VAE_Dec --> Image["Generated Image"]:::external

    click Demo "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/demo.ipynb"
    click NoiseNB "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/add_noise.ipynb"
    click Pipeline "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/pipeline.py"
    click ModelLoader "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/model_loader.py"
    click ModelConverter "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/model_converter.py"
    click TextEncoder "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/clip.py"
    click UNet "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/diffusion.py"
    click Scheduler "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/ddpm.py"
    click Attention "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/attention.py"
    click VAE_Enc "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/encoder.py"
    click VAE_Dec "https://github.com/hkproj/pytorch-stable-diffusion/blob/main/sd/decoder.py"

    classDef external fill:#dddddd,stroke:#333,stroke-width:1px
    classDef model fill:#bbdefb,stroke:#1e88e5,stroke-width:1px
    classDef orchestrator fill:#c8e6c9,stroke:#388e3c,stroke-width:1px
    classDef ui fill:#fff9c4,stroke:#fbc02d,stroke-width:1px
```
