How dyedd/deepspeed-diffusers works
Project type and purpose
• This is an open-source command-line tool & library for training and sampling diffusion models using Deepspeed, Huggingface Diffusers, and PEFT.
• It provides native integration with DeepSpeed (ZeRO-1/2/3), supports LoRA finetuning, EMA, custom datasets, Slurm scripts, and optional WandB logging.
File structure
• Top-level code:
• train_text_to_image.py – main training entry point
• test_text_to_image.py – main sampling/validation entry
• utils.py – helper functions (e.g., building DeepSpeed config)
• Configuration:
• cfg/default.json – all hyperparameters, paths, DeepSpeed settings
• Scripts (CLI wrappers & Slurm):
• scripts/*.sh and *.slurm – ready-made commands to launch training and sampling locally or via Slurm
• Experimental hooks:
• experimental/custom_dataset.py – example of extending data loading
• Requirements & metadata:
• requirements.txt – Python dependencies
• README*.md – usage, design rationale, feature list
Extract architecture insights from README
• Core libraries: Deepspeed, Diffusers (Unet2DConditionModel), Peft (LoRA)
• Two modes: full-model finetune vs. LoRA + Unet
• External integrations: Huggingface model hub, Slurm scheduler, optionally WandB
• Configuration-driven design: all parameters in cfg/default.json, parsed by utils.py
1. CLI & Script Layer
• Local shell scripts (train/test .sh)
• Slurm job scripts (train/test .slurm)
2. Configuration Manager
• cfg/default.json parsed by utils.deepspeed_config_from_args
3. Data Ingestion
• Dataset loader (Diffusers + custom_dataset.py)
• Preprocessing (center crop, flip)
4. Model Loader
• Pretrained model fetch (Huggingface hub or local path)
• Injection of LoRA modules (Peft)
5. Training Engine
• DeepSpeed engine wrapper
• ZeRO optimizer stages
• Mixed precision (FP16)
• EMA update (optional)
6. Checkpoint & Output
• periodic save of model weights & LoRA adapters
• output directory for images & logs
7. Logging & Monitoring
• WandB (online/offline)
• local console logs
8. Sampling / Validation Engine
• loading checkpoints
• Diffusion sampling loop
9. External Systems
• Slurm scheduler
• Huggingface Hub (for model weights)
• WandB service
B. Relationships & data flow
• Data flow: Configuration → Data Ingestion → Training Engine
• Model flow: Model Loader → Training Engine → Checkpoint → Sampling Engine
• Logging flow: Training Engine → Logging & Monitoring → External service
• Job orchestration: Slurm Scheduler invokes CLI Layer
• External fetch: Model Loader ↔ Huggingface Hub
C. Architectural patterns & principles
• Configuration-driven orchestration (all hyperparameters in JSON)
• Modular plugin points (utils.py for DeepSpeed config, experimental/custom_dataset.py)
• Separation of concerns: data, model, engine, logging, orchestration
• Reusable wrappers: unified train/test scripts for local vs. Slurm
D. Key technologies & frameworks
• Python 3.x
• Deepspeed (ZeRO-1/2/3)
• Diffusers (Unet2DConditionModel)
• Peft (LoRA)
• Slurm (batch scheduling)
• WandB (experiment tracking)
• Huggingface Datasets & Hub