# yifan123/flow_grpo architecture

> Architecture diagram of the GitHub repository yifan123/flow_grpo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yifan123/flow_grpo
- Repository: https://github.com/yifan123/flow_grpo

## Overview

##### Project type and purpose
- Flow-GRPO is a research-grade Python package for training flow-matching diffusion models via online reinforcement learning.
- It integrates multiple reward models (GenEval, OCR, PickScore, etc.) and uses distributed training (single-node, multi-node, Deepspeed/Accelerate) to optimize image generation.

##### File structure
Top-level directories:
- config/ – hyperparameter definitions for different compute environments (base.py, dgx.py)
- dataset/ – prompt metadata loaders and preprocessing scripts for each reward task (geneval, ocr, pickscore)
- flow_grpo/ – core library:
– RL engine (qwenvl.py, rewards.py, stat_tracking.py)
– Reward-scorer modules (aesthetic_scorer.py, imagereward_scorer.py, ocr.py, pickscore_scorer.py, etc.)
– diffusers_patch/ – custom patches to Hugging Face diffusers pipelines for logging log-probs and DreamBooth LoRA training
– prompts.py, ema.py (exponential moving average), assets/ (class lists, pretrained checkpoints)
- scripts/ – training orchestration:
– single_node/ and multi_node/ bash scripts for launching runs
– accelerate_configs/ YAMLs for Deepspeed/Accelerate setups
– train_sd3.py – entry point for high-level Python training launch
- setup.py – makes it an installable package

##### README for additional insights
- Describes three tasks (GenEval, OCR, PickScore) with separate reward-server dependencies
- Outlines installation, multi-reward weightings, and hyperparameter formulas
- Points to external reward-server for GenEval and the need for paddle-ocr for OCR reward

1. Configuration Service
– Reads config/base.py or config/dgx.py
2. Training Orchestrator (scripts + train_sd3.py)
3. Dataset Loader
– geneval, ocr, pickscore metadata readers & preprocessors
4. RL Engine (flow_grpo core)
– Environment & policy (qwenvl.py)
– Reward aggregator (rewards.py)
– Model update & EMA (ema.py)
– Stat tracking (stat_tracking.py)
5. Diffusion Pipeline (patched diffusers)
– sd3_sde_with_logprob, sd3_pipeline_with_logprob
6. Reward Scorers (local)
– aesthetic_scorer, imagereward_scorer, pickscore_scorer, ocr.py
7. External Reward Servers
– GenEval reward-server (remote endpoint)
– DeQA or other remote services
8. Compute Infrastructure
– Single-node/Multi-node cluster
– Deepspeed/Accelerate
9. Artifact Storage
– Model checkpoints (Hugging Face), logs, metrics

1. Orchestrator → Configuration Service (load hyperparameters)
2. Orchestrator → Dataset Loader (fetch prompts & metadata)
3. RL Engine ↔ Diffusion Pipeline (generate candidate images & compute log-probs)
4. RL Engine → Reward Scorers (send images)
- Local scorers (in-process function calls)
- External servers (HTTP/gRPC calls)
5. Reward Scorers → RL Engine (return reward values)
6. RL Engine → Compute Infrastructure (gradient steps via Deepspeed/Accelerate)
7. RL Engine → Artifact Storage (save checkpoints, logs)
8. Stat Tracking → Artifact Storage (metrics dashboards)

c. Architectural patterns & principles:
- Modular, plugin-based reward interface – new reward models can be “plugged in.”
- Clean separation between orchestration (scripts), configuration, data loading, core RL logic, and scoring.
- Use of distributed training frameworks (Deepspeed/Accelerate) abstracts hardware.
- Patches to standard libraries (diffusers) to expose additional signals (log-probs).

d. Key technologies/frameworks:
- Python 3.10, Conda environments
- Hugging Face diffusers + custom patches
- ddpo-pytorch design for RL training loops
- Deepspeed/Accelerate for multi-GPU/multi-node
- PaddleOCR for OCR reward
- External reward-server (GenEval)

– Blue rectangles: core components (config, orchestrator, RL engine)
– Green ovals: data sources (dataset loaders)
– Orange hexagons: reward scorers (local vs. remote)
– Gray clouds: external services (GenEval server)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Service
    subgraph "Configuration Service" 
        direction TB
        ConfigBase["base.py"]:::core
        ConfigDGX["dgx.py"]:::core
    end
    click ConfigBase "https://github.com/yifan123/flow_grpo/blob/main/config/base.py"
    click ConfigDGX "https://github.com/yifan123/flow_grpo/blob/main/config/dgx.py"

    %% Training Orchestrator
    subgraph "Training Orchestrator" 
        direction TB
        TrainSD3["train_sd3.py"]:::core
        ScriptsSingle["single_node/"]:::core
        ScriptsMulti["multi_node/"]:::core
        ScriptsAccel["accelerate_configs/"]:::core
    end
    click TrainSD3 "https://github.com/yifan123/flow_grpo/blob/main/scripts/train_sd3.py"
    click ScriptsSingle "https://github.com/yifan123/flow_grpo/tree/main/scripts/single_node/"
    click ScriptsMulti "https://github.com/yifan123/flow_grpo/tree/main/scripts/multi_node/"
    click ScriptsAccel "https://github.com/yifan123/flow_grpo/tree/main/scripts/accelerate_configs/"

    %% Dataset Loader
    subgraph "Dataset Loader" 
        direction TB
        subgraph Geneval
            direction TB
            GenevalDir["geneval/"]:::data
        end
        subgraph OCR
            direction TB
            OCRDir["ocr/"]:::data
        end
        subgraph PickScore
            direction TB
            PickDir["pickscore/"]:::data
        end
    end
    click GenevalDir "https://github.com/yifan123/flow_grpo/tree/main/dataset/geneval/"
    click OCRDir "https://github.com/yifan123/flow_grpo/tree/main/dataset/ocr/"
    click PickDir "https://github.com/yifan123/flow_grpo/tree/main/dataset/pickscore/"

    %% Core RL Engine
    subgraph "RL Engine" 
        direction TB
        QWEnv["qwenvl.py"]:::core
        Rewards["rewards.py"]:::core
        StatTrack["stat_tracking.py"]:::core
        EMA["ema.py"]:::core
        Prompts["prompts.py"]:::core
    end
    click QWEnv "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/qwenvl.py"
    click Rewards "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/rewards.py"
    click StatTrack "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/stat_tracking.py"
    click EMA "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/ema.py"
    click Prompts "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/prompts.py"

    %% Diffusion Pipeline
    subgraph "Diffusion Pipeline" 
        direction TB
        SD3Pipe["sd3_pipeline_with_logprob.py"]:::core
        SD3SDE["sd3_sde_with_logprob.py"]:::core
        DreamBooth["train_dreambooth_lora_sd3.py"]:::core
    end
    click SD3Pipe "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/diffusers_patch/sd3_pipeline_with_logprob.py"
    click SD3SDE "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/diffusers_patch/sd3_sde_with_logprob.py"
    click DreamBooth "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/diffusers_patch/train_dreambooth_lora_sd3.py"

    %% Reward Scorers (Local)
    subgraph "Local Reward Scorers" 
        direction TB
        Aesthetic["aesthetic_scorer.py"]:::scorer
        ImageReward["imagereward_scorer.py"]:::scorer
        OCRScorer["ocr.py"]:::scorer
        PickScoreScorer["pickscore_scorer.py"]:::scorer
    end
    click Aesthetic "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/aesthetic_scorer.py"
    click ImageReward "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/imagereward_scorer.py"
    click OCRScorer "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/ocr.py"
    click PickScoreScorer "https://github.com/yifan123/flow_grpo/blob/main/flow_grpo/pickscore_scorer.py"

    %% External Reward Servers
    subgraph "External Reward Servers" 
        direction TB
        GenEvalSrv["GenEval Server"]:::external
        DeQaSrv["DeQA Server"]:::external
    end

    %% Compute Infrastructure
    subgraph "Compute Infrastructure" 
        direction TB
        Accelerate["Deepspeed/Accelerate"]:::core
        SingleNode["Single-node Cluster"]:::core
        MultiNode["Multi-node Cluster"]:::core
        DSConfig1["deepspeed_zero1.yaml"]:::core
        DSConfig2["deepspeed_zero2.yaml"]:::core
        DSConfig3["deepspeed_zero3.yaml"]:::core
        MultiGPU["multi_gpu.yaml"]:::core
        MultiNodeYAML["multi_node.yaml"]:::core
    end
    click DSConfig1 "https://github.com/yifan123/flow_grpo/blob/main/scripts/accelerate_configs/deepspeed_zero1.yaml"
    click DSConfig2 "https://github.com/yifan123/flow_grpo/blob/main/scripts/accelerate_configs/deepspeed_zero2.yaml"
    click DSConfig3 "https://github.com/yifan123/flow_grpo/blob/main/scripts/accelerate_configs/deepspeed_zero3.yaml"
    click MultiGPU "https://github.com/yifan123/flow_grpo/blob/main/scripts/accelerate_configs/multi_gpu.yaml"
    click MultiNodeYAML "https://github.com/yifan123/flow_grpo/blob/main/scripts/accelerate_configs/multi_node.yaml"

    %% Artifact Storage
    subgraph "Artifact Storage" 
        direction TB
        Assets["assets/"]:::storage
        SetupPkg["setup.py"]:::storage
    end
    click Assets "https://github.com/yifan123/flow_grpo/tree/main/flow_grpo/assets/"
    click SetupPkg "https://github.com/yifan123/flow_grpo/blob/main/setup.py"

    %% Connections
    TrainSD3 -->|"loads hyperparameters"| ConfigBase
    TrainSD3 -->|"loads hyperparameters"| ConfigDGX
    TrainSD3 -->|"fetches prompts & metadata"| GenevalDir
    TrainSD3 -->|"fetches prompts & metadata"| OCRDir
    TrainSD3 -->|"fetches prompts & metadata"| PickDir

    QWEnv <-->|"generate images & log-probs"| SD3Pipe
    QWEnv <-->|"generate images & log-probs"| SD3SDE
    SD3Pipe -->|"model update"| EMA
    SD3SDE -->|"model update"| EMA

    QWEnv -->|"send images"| Aesthetic
    QWEnv -->|"send images"| ImageReward
    QWEnv -->|"send images"| OCRScorer
    QWEnv -->|"send images"| PickScoreScorer
    QWEnv -->|"send images"| GenEvalSrv
    QWEnv -->|"send images"| DeQaSrv

    Aesthetic -->|"reward values"| QWEnv
    ImageReward -->|"reward values"| QWEnv
    OCRScorer -->|"reward values"| QWEnv
    PickScoreScorer -->|"reward values"| QWEnv
    GenEvalSrv -->|"reward values"| QWEnv
    DeQaSrv -->|"reward values"| QWEnv

    QWEnv -->|"gradient steps"| Accelerate
    Accelerate -->|"uses config"| DSConfig1
    Accelerate -->|"uses config"| DSConfig2
    Accelerate -->|"uses config"| DSConfig3
    Accelerate -->|"uses config"| MultiGPU
    Accelerate -->|"uses config"| MultiNodeYAML

    QWEnv -->|"save checkpoints & logs"| Assets
    StatTrack -->|"write metrics"| Assets

    %% Styles
    classDef core fill:#bbdefb,stroke:#1e88e5,color:#0d47a1
    classDef data fill:#c8e6c9,stroke:#43a047,color:#1b5e20
    classDef scorer fill:#ffe0b2,stroke:#fb8c00,color:#e65100
    classDef storage fill:#e1bee7,stroke:#8e24aa,color:#4a148c
    classDef external fill:#eceff1,stroke:#90a4ae,color:#37474f
```
