# kvablack/ddpo-pytorch architecture

> Architecture diagram of the GitHub repository kvablack/ddpo-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kvablack/ddpo-pytorch
- Repository: https://github.com/kvablack/ddpo-pytorch

## Overview

##### Project Type and Purpose
- This is a research-oriented machine-learning training pipeline written in Python.
- It implements Denoising Diffusion Policy Optimization (DDPO) on top of HuggingFace’s Stable Diffusion, with optional Low-Rank Adaptation (LoRA).
- Its goal is to generate images from text prompts, score them with a reward function, and finetune the diffusion model via policy gradients.

##### File Structure
- Top-level:
- config/ – Python config files defining hyperparameters and experiment presets (base.py, dgx.py)
- ddpo_pytorch/ – Core library package:
– aesthetic_scorer.py, rewards.py – reward-function modules
– prompts.py – prompt generation logic
– stat_tracking.py – logging and metric tracking (e.g., to Weights & Biases)
– diffusers_patch/ – custom patches to diffusers pipelines for log-prob support
– assets/ – static files (class lists, pretrained LoRA weights, activity vocab)
- scripts/train.py – entrypoint that uses `accelerate` to orchestrate training
- setup.py – package installer script
- Configuration files and external tool integration:
- Uses HuggingFace Diffusers, Accelerate, optional WandB, and PyTorch.

##### README for Additional Insights
- Training flow:
1. Load config → instantiate model (Stable Diffusion + optional LoRA adapters)
2. Loop per epoch:
a. Generate prompts via prompt_fn
b. Sample images using the diffusion model
c. Compute rewards via reward_fn
d. Perform policy gradient update steps with gradient accumulation
e. Log metrics and checkpoints
- Supports multi-GPU via Accelerate; hyperparameters define effective batch sizes.
- Has two config presets (base for quick tests, dgx for 8×A100).
- External inference service for LLaVA if chosen.

##### Main Components and Their Interactions
a. Prompt Generator (ddpo_pytorch/prompts.py)
b. Diffusion Model + LoRA Adapter (diffusers + ddpo_pytorch/diffusers_patch)
c. Sampling Procedure (inside scripts/train.py orchestrated by Accelerate)
d. Reward Evaluator (ddpo_pytorch/rewards.py, aesthetic_scorer.py)
e. DDPO Optimizer Loop (policy gradient updates, config/train parameters)
f. Metrics & Logging (ddpo_pytorch/stat_tracking.py → WandB or console)
g. Configuration Manager (config/base.py, config/dgx.py)
h. External Services: HuggingFace Hub, accelerate launcher, optional WandB, optional LLaVA server

##### Architectural Patterns and Design Principles
- Modular package layout separating core logic (ddpo_pytorch/) from entrypoint (scripts/) and configuration (config/).
- Plugin-style reward and prompt functions for extensibility.
- Use of patches rather than forking the entire Diffusers library.
- Clear separation of concerns: data generation, reward evaluation, optimization, logging.
- Multi-GPU abstraction via HuggingFace Accelerate.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Manager
    subgraph "Configuration Manager"
        direction TB
        base_cfg["config/base.py"]:::config
        dgx_cfg["config/dgx.py"]:::config
    end

    %% Core ML Components
    subgraph "Core Pipeline"
        direction TB
        prompts["Prompt Generator\n(ddpo_pytorch/prompts.py)"]:::core
        subgraph "Diffusion Model\n+ LoRA Adapter"
            direction TB
            sd_model["Stable Diffusion\n(HuggingFace Diffusers)"]:::core
            subgraph "Diffusers Patch"
                direction TB
                patch1["ddim_with_logprob.py"]:::core
                patch2["pipeline_with_logprob.py"]:::core
            end
        end
        sampler["Sampling Procedure\n(scripts/train.py)"]:::core
        rewards["Reward Evaluator\n(ddpo_pytorch/rewards.py)"]:::core
        aesthetic["Aesthetic Scorer\n(ddpo_pytorch/aesthetic_scorer.py)"]:::core
        optimizer["DDPO Policy Gradient\nLoop\n(scripts/train.py)"]:::core
        logger["Metrics & Logging\n(ddpo_pytorch/stat_tracking.py)"]:::core
    end

    %% Static Assets
    assets["Static Assets\n(ddpo_pytorch/assets/)"]:::config

    %% External Services
    subgraph "External Services"
        direction TB
        hf_hub["HuggingFace Hub"]:::external
        wandb["Weights & Biases"]:::external
        llava["LLaVA Server (Optional)"]:::external
    end

    %% Accelerate Multi-GPU Boundary
    subgraph "Accelerate\n(Multi-GPU)"
        direction TB
        sampler
        optimizer
    end

    %% Data Flow
    base_cfg -->|select preset| sampler
    dgx_cfg -->|select preset| sampler
    sampler --> prompts
    prompts --> sd_model
    sd_model --> patch1
    sd_model --> patch2
    patch2 --> sampler
    sampler -->|"generated images"| rewards
    rewards --> aesthetic
    aesthetic --> optimizer
    optimizer --> sd_model
    optimizer --> logger
    assets --> sd_model
    hf_hub --> sd_model
    logger --> wandb
    sampler --> llava

    %% Click Events
    click prompts "https://github.com/kvablack/ddpo-pytorch/blob/main/ddpo_pytorch/prompts.py"
    click patch1 "https://github.com/kvablack/ddpo-pytorch/blob/main/ddpo_pytorch/diffusers_patch/ddim_with_logprob.py"
    click patch2 "https://github.com/kvablack/ddpo-pytorch/blob/main/ddpo_pytorch/diffusers_patch/pipeline_with_logprob.py"
    click sampler "https://github.com/kvablack/ddpo-pytorch/blob/main/scripts/train.py"
    click optimizer "https://github.com/kvablack/ddpo-pytorch/blob/main/scripts/train.py"
    click rewards "https://github.com/kvablack/ddpo-pytorch/blob/main/ddpo_pytorch/rewards.py"
    click aesthetic "https://github.com/kvablack/ddpo-pytorch/blob/main/ddpo_pytorch/aesthetic_scorer.py"
    click logger "https://github.com/kvablack/ddpo-pytorch/blob/main/ddpo_pytorch/stat_tracking.py"
    click base_cfg "https://github.com/kvablack/ddpo-pytorch/blob/main/config/base.py"
    click dgx_cfg "https://github.com/kvablack/ddpo-pytorch/blob/main/config/dgx.py"
    click assets "https://github.com/kvablack/ddpo-pytorch/tree/main/ddpo_pytorch/assets/"

    %% Styles
    classDef core fill:#D6EAF8,stroke:#2980B9,color:#154360
    classDef config fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef external fill:#FAD7A0,stroke:#D68910,color:#7E5109
```
