# kohya-ss/sd-scripts architecture

> Architecture diagram of the GitHub repository kohya-ss/sd-scripts, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kohya-ss/sd-scripts
- Repository: https://github.com/kohya-ss/sd-scripts

## Overview

##### Project’s main purpose and type
- It is an ML toolkit for Stable Diffusion that includes both training and inference components. The repository covers multiple training methods (DreamBooth, LoRA training, textual inversion, etc.), model conversion scripts, and utility functions supporting preprocessing, tag generation, and image manipulation.
- The README emphasizes training pipelines, integration with external libraries (PyTorch, xformers, bitsandbytes, DeepSpeed, etc.), and different configuration files. This tells you that the system is modular with dedicated pipelines for each training/inference strategy.

##### Break the system into major components drawn as discrete blocks
- Training Pipelines:
- Training scripts such as “train_network.py”, “sdxl_train.py”, “train_controlnet.py”, “train_db.py”, “train_textual_inversion.py”, etc.
- That these scripts depend heavily on ML libraries (PyTorch, accelerate, DeepSpeed, etc.) and use supporting utilities from the “library” and “networks” directories.
- That these pipelines are further subdivided into different training types (DreamBooth, LoRA variants, SDXL support, etc.).
- Generation/Inferences:
- Scripts like “gen_img.py”, “gen_img_diffusers.py”, “sdxl_gen_img.py”, and “sdxl_minimal_inference.py” as the generation layer that loads trained models to produce images.
- That these often reuse similar underlying libraries and device utilities.
- Fine-tuning and Preprocessing:
- The “finetune” directory and its submodules (including BLIP-related scripts and caption processing code) are in charge of additional data preparation and fine-tuning operations.
- This part interacts with the training pipelines.
- Core Libraries and Utilities:
- The “library” directory serves as the backbone with modules handling model utilities (e.g., “model_util.py”, “train_util.py”), custom training functions, device utilities, and integration with external frameworks (HuggingFace utilities, deepspeed_utils, etc.).
- The “networks” folder encapsulates modules for various LoRA methods and other network modifications.
- Tools and Auxiliary Scripts:
- The “tools” folder holds utility scripts for tasks like latent caching (“cache_latents.py”), canny edge detection (“canny.py”), file conversion, metadata display, and other image processing operations.
- OS/Dependency Specific Modules:
- The “bitsandbytes_windows” folder contains Windows-specific adaptations ensuring compatibility with the bitsandbytes library. This detail might be noted as an environment-dependent component.
- Documentation and Configuration:

##### Relationships and interactions
- Data flow should be clearly indicated: input data (images, text prompts, captions) flows into the training pipelines (sometimes via preprocessing tools), then the trained models are stored and later used by the generation/inference pipeline.
- That there is a configuration/metadata exchange – for example, dataset settings from the .toml files (as described within the docs) inform how the training scripts process data. This can be shown as a data/configuration store that several components refer to.

##### Detail important architectural patterns and design principles
- Modular design: Each training script is independent yet relies on a common set of utility modules – highlight inter-module reuse.
- Separation of concerns: Training, generation, and utility operations are distinct but interrelated modules.
- Extensibility: The “networks” folder shows how the system has been extended to support multiple types of LoRA and additional network modifications.
- Environment-aware configuration: The bitsandbytes_windows folder and instructions for different CUDA versions illustrate that configuration is critical and must be clearly noted.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Dependencies
    E1["External Dependencies"]:::external

    %% Training Pipelines Subgraph
    subgraph "Training Pipelines"
        DB["DreamBooth Training"]:::training
        NT["LoRA & Network Training"]:::training
        TI["Textual Inversion Training"]:::training
        TIX["Textual Inversion (XTI)"]:::training
        subgraph "SDXL Training"
            S1["SDXL Main Training"]:::training
            S2["SDXL ControlNet LLLite"]:::training
            S3["SDXL ControlNet LLLite Old"]:::training
            S4["SDXL Network Training"]:::training
            S5["SDXL Textual Inversion"]:::training
        end
    end

    %% Generation/Inferences Subgraph
    subgraph "Generation/Inferences"
        G1["Standard Image Generation"]:::inference
        GD["Standard Diffusers Generation"]:::inference
        G2["SDXL Image Generation"]:::inference
        G3["SDXL Minimal Inference"]:::inference
    end

    %% Fine-tuning and Preprocessing Subgraph
    subgraph "Fine-tuning/Preprocessing"
        F1["Fine-tuning Script"]:::inference
        F2["Finetune Module"]:::inference
    end

    %% Core Libraries and Utilities Subgraph
    subgraph "Core Libraries & Utilities"
        C1["Core Libraries"]:::core
        D1["Device Utils"]:::core
        D2["DeepSpeed Utils"]:::core
        D3["HuggingFace Util"]:::core
    end

    %% Networks and Model Modifications Subgraph
    subgraph "Networks & Model Modifications"
        N1["Network Extensions"]:::core
        N2["LoRA Implementation"]:::core
        N3["DYLoRA Implementation"]:::core
    end

    %% Tools and Auxiliary Scripts Subgraph
    subgraph "Tools & Auxiliary Scripts"
        TOLS["Utility Tools"]:::util
    end

    %% OS/Dependency Specific Subgraph
    subgraph "OS/Dependency Specific"
        O1["Windows BitsAndBytes"]:::util
    end

    %% Documentation & Configuration Subgraph
    subgraph "Documentation & Configuration"
        DOC["Documentation & Config"]:::util
    end

    %% Data Flow Arrows
    %% Fine-tuning/Preprocessing to Training Pipelines
    F2 ---|"preprocess"| DB
    F2 ---|"preprocess"| TI

    %% Training Pipelines dependencies to Core Libraries
    DB ---|"calls"| C1
    NT ---|"calls"| C1
    TI ---|"calls"| C1
    TIX ---|"calls"| C1
    S1 ---|"calls"| C1
    S2 ---|"calls"| C1
    S3 ---|"calls"| C1
    S4 ---|"calls"| C1
    S5 ---|"calls"| C1

    %% Training Pipelines dependency to Networks & Model Modifications
    NT ---|"integrates"| N1
    NT ---|"integrates"| N2
    NT ---|"integrates"| N3

    %% Generation/Inferences dependencies to Core Libraries
    G1 ---|"loads"| C1
    GD ---|"loads"| C1
    G2 ---|"loads"| C1
    G3 ---|"loads"| C1

    %% Core Libraries integration with Device Utilities
    C1 ---|"provides"| D1
    C1 ---|"provides"| D2
    C1 ---|"provides"| D3

    %% External Dependencies used by Training & Inference
    DB ---|"uses"| E1
    NT ---|"uses"| E1
    TI ---|"uses"| E1
    S1 ---|"uses"| E1
    G1 ---|"uses"| E1
    GD ---|"uses"| E1
    G2 ---|"uses"| E1
    G3 ---|"uses"| E1

    %% External Dependencies and OS Specific
    E1 ---|"compatibility"| O1

    %% Tools and Documentation support
    TOLS ---|"assists"| C1
    DOC ---|"informs"| C1

    %% Click Events for Training Pipelines
    click DB "https://github.com/kohya-ss/sd-scripts/blob/main/train_db.py"
    click NT "https://github.com/kohya-ss/sd-scripts/blob/main/train_network.py"
    click TI "https://github.com/kohya-ss/sd-scripts/blob/main/train_textual_inversion.py"
    click TIX "https://github.com/kohya-ss/sd-scripts/blob/main/train_textual_inversion_XTI.py"
    click S1 "https://github.com/kohya-ss/sd-scripts/blob/main/sdxl_train.py"
    click S2 "https://github.com/kohya-ss/sd-scripts/blob/main/sdxl_train_control_net_lllite.py"
    click S3 "https://github.com/kohya-ss/sd-scripts/blob/main/sdxl_train_control_net_lllite_old.py"
    click S4 "https://github.com/kohya-ss/sd-scripts/blob/main/sdxl_train_network.py"
    click S5 "https://github.com/kohya-ss/sd-scripts/blob/main/sdxl_train_textual_inversion.py"

    %% Click Events for Generation/Inferences
    click G1 "https://github.com/kohya-ss/sd-scripts/blob/main/gen_img.py"
    click GD "https://github.com/kohya-ss/sd-scripts/blob/main/gen_img_diffusers.py"
    click G2 "https://github.com/kohya-ss/sd-scripts/blob/main/sdxl_gen_img.py"
    click G3 "https://github.com/kohya-ss/sd-scripts/blob/main/sdxl_minimal_inference.py"

    %% Click Events for Fine-tuning and Preprocessing
    click F1 "https://github.com/kohya-ss/sd-scripts/blob/main/fine_tune.py"
    click F2 "https://github.com/kohya-ss/sd-scripts/tree/main/finetune"

    %% Click Events for Core Libraries and Utilities
    click C1 "https://github.com/kohya-ss/sd-scripts/tree/main/library"
    click D1 "https://github.com/kohya-ss/sd-scripts/blob/main/library/device_utils.py"
    click D2 "https://github.com/kohya-ss/sd-scripts/blob/main/library/deepspeed_utils.py"
    click D3 "https://github.com/kohya-ss/sd-scripts/blob/main/library/huggingface_util.py"

    %% Click Events for Networks and Model Modifications
    click N1 "https://github.com/kohya-ss/sd-scripts/tree/main/networks"
    click N2 "https://github.com/kohya-ss/sd-scripts/blob/main/networks/lora.py"
    click N3 "https://github.com/kohya-ss/sd-scripts/blob/main/networks/dylora.py"

    %% Click Events for Tools and Auxiliary Scripts
    click TOLS "https://github.com/kohya-ss/sd-scripts/tree/main/tools"

    %% Click Events for OS/Dependency Specific Modules
    click O1 "https://github.com/kohya-ss/sd-scripts/tree/main/bitsandbytes_windows"

    %% Click Events for Documentation & Configuration
    click DOC "https://github.com/kohya-ss/sd-scripts/tree/main/docs"

    %% Styles
    classDef training fill:#FFCCCC,stroke:#AA0000,stroke-width:2px;
    classDef inference fill:#CCE5FF,stroke:#004085,stroke-width:2px;
    classDef core fill:#FFF3CD,stroke:#856404,stroke-width:2px;
    classDef util fill:#E2E3E5,stroke:#6C757D,stroke-width:2px;
    classDef external fill:#D1ECF1,stroke:#0C5460,stroke-width:2px;
```
