How kvablack/ddpo-pytorch works
Project Type and Purpose
• This is a research-oriented machine-learning training pipeline written in Python.
• It implements Denoising Diffusion Policy Optimization (DDPO) on top of HuggingFace’s Stable Diffusion, with optional Low-Rank Adaptation (LoRA).
• Its goal is to generate images from text prompts, score them with a reward function, and finetune the diffusion model via policy gradients.
File Structure
• Top-level:
• config/ – Python config files defining hyperparameters and experiment presets (base.py, dgx.py)
• ddpo_pytorch/ – Core library package:
– aesthetic_scorer.py, rewards.py – reward-function modules
– prompts.py – prompt generation logic
– stat_tracking.py – logging and metric tracking (e.g., to Weights & Biases)
– diffusers_patch/ – custom patches to diffusers pipelines for log-prob support
– assets/ – static files (class lists, pretrained LoRA weights, activity vocab)
• scripts/train.py – entrypoint that uses accelerate to orchestrate training
• setup.py – package installer script
• Configuration files and external tool integration:
• Uses HuggingFace Diffusers, Accelerate, optional WandB, and PyTorch.
README for Additional Insights
• Training flow:
1. Load config → instantiate model (Stable Diffusion + optional LoRA adapters)
2. Loop per epoch:
a. Generate prompts via prompt_fn
b. Sample images using the diffusion model
c. Compute rewards via reward_fn
d. Perform policy gradient update steps with gradient accumulation
e. Log metrics and checkpoints
• Supports multi-GPU via Accelerate; hyperparameters define effective batch sizes.
• Has two config presets (base for quick tests, dgx for 8×A100).
• External inference service for LLaVA if chosen.
Main Components and Their Interactions
a. Prompt Generator (ddpo_pytorch/prompts.py)
b. Diffusion Model + LoRA Adapter (diffusers + ddpo_pytorch/diffusers_patch)
c. Sampling Procedure (inside scripts/train.py orchestrated by Accelerate)
d. Reward Evaluator (ddpo_pytorch/rewards.py, aesthetic_scorer.py)
e. DDPO Optimizer Loop (policy gradient updates, config/train parameters)
f. Metrics & Logging (ddpo_pytorch/stat_tracking.py → WandB or console)
g. Configuration Manager (config/base.py, config/dgx.py)
h. External Services: HuggingFace Hub, accelerate launcher, optional WandB, optional LLaVA server
Architectural Patterns and Design Principles
• Modular package layout separating core logic (ddpo_pytorch/) from entrypoint (scripts/) and configuration (config/).
• Plugin-style reward and prompt functions for extensibility.
• Use of patches rather than forking the entire Diffusers library.
• Clear separation of concerns: data generation, reward evaluation, optimization, logging.
• Multi-GPU abstraction via HuggingFace Accelerate.