# dyedd/deepspeed-diffusers architecture

> Architecture diagram of the GitHub repository dyedd/deepspeed-diffusers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dyedd/deepspeed-diffusers
- Repository: https://github.com/dyedd/deepspeed-diffusers

## Overview

##### Project type and purpose
- This is an open-source command-line tool & library for training and sampling diffusion models using Deepspeed, Huggingface Diffusers, and PEFT.
- It provides native integration with DeepSpeed (ZeRO-1/2/3), supports LoRA finetuning, EMA, custom datasets, Slurm scripts, and optional WandB logging.

##### File structure
- Top-level code:
- train_text_to_image.py – main training entry point
- test_text_to_image.py – main sampling/validation entry
- utils.py – helper functions (e.g., building DeepSpeed config)
- Configuration:
- cfg/default.json – all hyperparameters, paths, DeepSpeed settings
- Scripts (CLI wrappers & Slurm):
- scripts/*.sh and *.slurm – ready-made commands to launch training and sampling locally or via Slurm
- Experimental hooks:
- experimental/custom_dataset.py – example of extending data loading
- Requirements & metadata:
- requirements.txt – Python dependencies
- README*.md – usage, design rationale, feature list

##### Extract architecture insights from README
- Core libraries: Deepspeed, Diffusers (Unet2DConditionModel), Peft (LoRA)
- Two modes: full-model finetune vs. LoRA + Unet
- External integrations: Huggingface model hub, Slurm scheduler, optionally WandB
- Configuration-driven design: all parameters in cfg/default.json, parsed by utils.py

1. CLI & Script Layer
- Local shell scripts (train/test .sh)
- Slurm job scripts (train/test .slurm)
2. Configuration Manager
- cfg/default.json parsed by utils.deepspeed_config_from_args
3. Data Ingestion
- Dataset loader (Diffusers + custom_dataset.py)
- Preprocessing (center crop, flip)
4. Model Loader
- Pretrained model fetch (Huggingface hub or local path)
- Injection of LoRA modules (Peft)
5. Training Engine
- DeepSpeed engine wrapper
- ZeRO optimizer stages
- Mixed precision (FP16)
- EMA update (optional)
6. Checkpoint & Output
- periodic save of model weights & LoRA adapters
- output directory for images & logs
7. Logging & Monitoring
- WandB (online/offline)
- local console logs
8. Sampling / Validation Engine
- loading checkpoints
- Diffusion sampling loop
9. External Systems
- Slurm scheduler
- Huggingface Hub (for model weights)
- WandB service

B. Relationships & data flow
- Data flow: Configuration → Data Ingestion → Training Engine
- Model flow: Model Loader → Training Engine → Checkpoint → Sampling Engine
- Logging flow: Training Engine → Logging & Monitoring → External service
- Job orchestration: Slurm Scheduler invokes CLI Layer
- External fetch: Model Loader ↔ Huggingface Hub

C. Architectural patterns & principles
- Configuration-driven orchestration (all hyperparameters in JSON)
- Modular plugin points (utils.py for DeepSpeed config, experimental/custom_dataset.py)
- Separation of concerns: data, model, engine, logging, orchestration
- Reusable wrappers: unified train/test scripts for local vs. Slurm

D. Key technologies & frameworks
- Python 3.x
- Deepspeed (ZeRO-1/2/3)
- Diffusers (Unet2DConditionModel)
- Peft (LoRA)
- Slurm (batch scheduling)
- WandB (experiment tracking)
- Huggingface Datasets & Hub

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        TrainSH["train_text_to_image.sh"]:::orchestration
        TestSH["test_text_to_image.sh"]:::orchestration
        TrainSL["train_text_to_image.slurm"]:::orchestration
        TestSL["test_text_to_image.slurm"]:::orchestration
    end

    %% Configuration Manager
    subgraph "Configuration Manager"
        direction TB
        CFG["cfg/default.json"]:::core
        Utils["utils.py"]:::core
    end

    %% Data Ingestion
    subgraph "Data Ingestion"
        direction TB
        CustomDS["custom_dataset.py"]:::core
    end

    %% Model Loader
    subgraph "Model Loader"
        direction TB
        TrainEntry["train_text_to_image.py"]:::core
        TestEntry["test_text_to_image.py"]:::core
    end

    %% Training Engine
    subgraph "Training Engine"
        direction TB
        DSEngine["DeepSpeed Engine"]:::framework
        ZeroOpt["ZeRO Optimizer"]:::framework
        MixedPrec["Mixed Precision"]:::framework
        EMA["EMA Update"]:::framework
    end

    %% Checkpoint & Output
    subgraph "Checkpoint & Output"
        direction TB
        Checkpoint["Model & LoRA Checkpoints"]:::core
        Outputs["Generated Images & Logs"]:::core
    end

    %% Logging & Monitoring
    subgraph "Logging & Monitoring"
        direction TB
        WandB["WandB Integration"]:::service
        ConsoleLog["Console Logs"]:::core
    end

    %% Sampling / Validation Engine
    subgraph "Sampling Engine"
        direction TB
        Sampling["Diffusion Sampling Loop"]:::framework
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        Slurm["Slurm Scheduler"]:::service
        HFHub["Huggingface Hub"]:::service
        Requirements["requirements.txt"]:::core
        README["README.md / README_en.md"]:::core
    end

    %% Connections
    Slurm -->|invokes| TrainSH
    Slurm -->|invokes| TestSH
    TrainSH -->|reads| CFG
    TestSH -->|reads| CFG
    CFG -->|parsed by| Utils
    Utils -->|provides config to| DSEngine
    CustomDS -->|hook into| TrainEntry
    TrainEntry -->|loads model| HFHub
    TrainEntry -->|injects LoRA via| Utils
    TrainEntry -->|initializes| DSEngine
    DSEngine -->|uses| ZeroOpt
    DSEngine -->|uses| MixedPrec
    DSEngine -->|optionally uses| EMA
    DSEngine -->|trains and outputs| Checkpoint
    DSEngine -->|outputs images| Outputs
    DSEngine -->|logs to| WandB
    DSEngine -->|logs to| ConsoleLog
    TestEntry -->|loads checkpoint| Checkpoint
    TestEntry -->|runs| Sampling
    Sampling -->|generates images| Outputs
    TrainEntry -->|integrates| WandB
    TrainEntry -->|console logs| ConsoleLog
    TrainEntry -->|uses| Utils

    TrainEntry -->|configured by| CFG
    TestEntry -->|configured by| CFG

    HFHub <-->|fetches model| TrainEntry

    %% Click Events
    click TrainSH "https://github.com/dyedd/deepspeed-diffusers/blob/main/scripts/train_text_to_image.sh"
    click TestSH "https://github.com/dyedd/deepspeed-diffusers/blob/main/scripts/test_text_to_image.sh"
    click TrainSL "https://github.com/dyedd/deepspeed-diffusers/blob/main/scripts/train_text_to_image.slurm"
    click TestSL "https://github.com/dyedd/deepspeed-diffusers/blob/main/scripts/test_text_to_image.slurm"
    click CFG "https://github.com/dyedd/deepspeed-diffusers/blob/main/cfg/default.json"
    click Utils "https://github.com/dyedd/deepspeed-diffusers/blob/main/utils.py"
    click CustomDS "https://github.com/dyedd/deepspeed-diffusers/blob/main/experimental/custom_dataset.py"
    click TrainEntry "https://github.com/dyedd/deepspeed-diffusers/blob/main/train_text_to_image.py"
    click TestEntry "https://github.com/dyedd/deepspeed-diffusers/blob/main/test_text_to_image.py"
    click Requirements "https://github.com/dyedd/deepspeed-diffusers/blob/main/requirements.txt"
    click README "https://github.com/dyedd/deepspeed-diffusers/blob/main/README.md"

    %% Styles
    classDef core fill:#bbf,stroke:#333,stroke-width:1px
    classDef framework fill:#9f6,stroke:#333,stroke-width:1px
    classDef orchestration fill:#f96,stroke:#333,stroke-width:1px
    classDef service fill:#c9f,stroke:#333,stroke-width:1px
```
