# segmind/distill-sd architecture

> Architecture diagram of the GitHub repository segmind/distill-sd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/segmind/distill-sd
- Repository: https://github.com/segmind/distill-sd

## Overview

##### Project Type and Purpose
- This is a machine‐learning pipeline/toolkit focused on distilling a large “teacher” Stable Diffusion model into smaller, faster “student” models (sd_small, sd_tiny).
- It provides scripts for data preparation, several training modes (distillation, checkpoint fine‐tuning, LoRA), inference, and model upload.

##### Main Architectural Components
A. External Resources
– Dataset (Hugging Face datasets, e.g., LAION‐art)
– Pretrained Teacher Model (e.g., SG161222/Realistic_Vision_V4.0)
– Hugging Face Model Hub (for pulling/pushing models)
B. Core Libraries & Frameworks
– Python
– PyTorch & Accelerate
– Hugging Face Diffusers (pipeline, scheduler)
C. Data Ingestion & Preparation
– data.py: downloads and preprocesses images and captions
D. Training Subsystems
1. Distillation Training (distill_training.py)
- Loads teacher and student U-nets
- Defines MSE losses at output and feature levels
- Scheduler, optimizer, gradient accumulation, checkpointing
2. Checkpoint Fine‐tuning (checkpoint_training.py)
- Loads an existing student checkpoint
- Continues training with EMA and diffusers scripts
3. LoRA Training (lora_training.py)
- Adds LoRA adapters to U-net/text encoder
- Trains low‐rank modules for style/concept fine‐tuning
4. Full T2I Training (trainT2I.py)
- Standard Stable Diffusion finetuning script (optional)
E. Inference Engine
– inference.py: loads distilled/LoRA/fine‐tuned model
– Configures scheduler and runs image generation
F. Model Management
– upload_model.py: pushes trained artifacts to Hugging Face hub
– Maintains config.json consistency, versioning
G. Assets & Documentation
– README.md: usage examples, hyperparameters, model stats

##### Relationships & Data Flow
- Data Ingestion → Training Subsystems
- Pretrained Teacher Model → Distillation Training
- Training Subsystems → Model Artifacts
– Checkpoints and final student model files
- Model Artifacts → Inference Engine → Generated Images
- Model Artifacts → upload_model.py → Hugging Face Hub
- Core Libraries underpin every script

##### Architectural Patterns & Principles
- Modular pipeline: each script is a distinct stage
- Reuse of common components (diffusers pipelines, scheduler)
- Separation of concerns: data preparation, distillation, fine‐tuning, inference, upload
- Pluggable configuration: command‐line flags for hyperparameters
- Dependency injection: paths to datasets and models provided at runtime

##### Detailed Capture of Essential Elements
– “Data Layer”, “Training Layer”, “Inference Layer”, “Model Management Layer”
- Call out loss components in the distillation block (output‐level, feature‐level)
- That distillation shares code from diffusers but adds custom loss functions
- Optional paths: full finetuning vs. checkpoint resume vs. LoRA branch
- Upload step as final block linking artifact storage to the Hub

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Libraries Layer
    subgraph "Core Libraries"
        CoreLibs["Python\nPyTorch + Accelerate\nHugging Face Diffusers"]:::libs
    end

    %% Data Layer
    subgraph "Data Layer"
        Dataset["Dataset\n(Hugging Face Datasets)"]:::storage
        TeacherModel["Pretrained Teacher Model\n(SG161222/Realistic_Vision_V4.0)"]:::storage
        DataPrep["data.py\n(Data Preparation)"]:::script
    end

    %% Training Layer
    subgraph "Training Subsystems"
        DistillTraining["Distillation Training\ndistill_training.py\n(MSE output & feature)"]:::script
        CheckpointTraining["Checkpoint Fine-tuning\ncheckpoint_training.py"]:::script
        LoraTraining["LoRA Training\nlora_training.py"]:::script
        FullT2I["Full Text-to-Image Training\ntrainT2I.py"]:::script
    end

    %% Model Artifacts
    Artifacts["Model Artifacts\n(Checkpoints & Student Models)"]:::storage

    %% Inference Layer
    subgraph "Inference Layer"
        Inference["Inference Engine\ninference.py"]:::script
        GeneratedImages["Generated Images"]:::storage
    end

    %% Model Management Layer
    subgraph "Model Management"
        Upload["Model Upload\nupload_model.py"]:::script
        Hub["Hugging Face Model Hub"]:::external
    end

    %% Assets & Documentation
    subgraph "Assets & Documentation"
        Assets["assets/\n(Images, Diagrams)"]:::assets
        Doc["README.md"]:::assets
    end

    %% Data Flow
    CoreLibs -.-> DataPrep
    CoreLibs -.-> DistillTraining
    CoreLibs -.-> CheckpointTraining
    CoreLibs -.-> LoraTraining
    CoreLibs -.-> FullT2I
    CoreLibs -.-> Inference
    CoreLibs -.-> Upload

    Dataset --> DataPrep
    TeacherModel --> DistillTraining
    DataPrep --> DistillTraining
    DataPrep --> CheckpointTraining
    DataPrep --> LoraTraining
    DataPrep --> FullT2I

    DistillTraining --> Artifacts
    CheckpointTraining --> Artifacts
    LoraTraining --> Artifacts
    FullT2I --> Artifacts

    Artifacts --> Inference
    Inference --> GeneratedImages

    Artifacts --> Upload
    Upload --> Hub

    %% Click Events
    click DataPrep "https://github.com/segmind/distill-sd/blob/master/data.py"
    click DistillTraining "https://github.com/segmind/distill-sd/blob/master/distill_training.py"
    click CheckpointTraining "https://github.com/segmind/distill-sd/blob/master/checkpoint_training.py"
    click LoraTraining "https://github.com/segmind/distill-sd/blob/master/lora_training.py"
    click FullT2I "https://github.com/segmind/distill-sd/blob/master/trainT2I.py"
    click Inference "https://github.com/segmind/distill-sd/blob/master/inference.py"
    click Upload "https://github.com/segmind/distill-sd/blob/master/upload_model.py"
    click Assets "https://github.com/segmind/distill-sd/tree/master/assets/"
    click Doc "https://github.com/segmind/distill-sd/blob/master/README.md"

    %% Styles
    classDef script fill:#f9f,stroke:#333,stroke-width:1px;
    classDef storage fill:#bbf,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    classDef external fill:#fbf,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2;
    classDef libs fill:#bfb,stroke:#333,stroke-width:1px;
    classDef assets fill:#ffd,stroke:#333,stroke-width:1px;
```
