How yifan123/flow_grpo works
Project type and purpose
• Flow-GRPO is a research-grade Python package for training flow-matching diffusion models via online reinforcement learning.
• It integrates multiple reward models (GenEval, OCR, PickScore, etc.) and uses distributed training (single-node, multi-node, Deepspeed/Accelerate) to optimize image generation.
File structure
Top-level directories:
• config/ – hyperparameter definitions for different compute environments (base.py, dgx.py)
• dataset/ – prompt metadata loaders and preprocessing scripts for each reward task (geneval, ocr, pickscore)
• flow_grpo/ – core library:
– RL engine (qwenvl.py, rewards.py, stat_tracking.py)
– Reward-scorer modules (aesthetic_scorer.py, imagereward_scorer.py, ocr.py, pickscore_scorer.py, etc.)
– diffusers_patch/ – custom patches to Hugging Face diffusers pipelines for logging log-probs and DreamBooth LoRA training
– prompts.py, ema.py (exponential moving average), assets/ (class lists, pretrained checkpoints)
• scripts/ – training orchestration:
– single_node/ and multi_node/ bash scripts for launching runs
– accelerate_configs/ YAMLs for Deepspeed/Accelerate setups
– train_sd3.py – entry point for high-level Python training launch
• setup.py – makes it an installable package
README for additional insights
• Describes three tasks (GenEval, OCR, PickScore) with separate reward-server dependencies
• Outlines installation, multi-reward weightings, and hyperparameter formulas
• Points to external reward-server for GenEval and the need for paddle-ocr for OCR reward
1. Configuration Service
– Reads config/base.py or config/dgx.py
2. Training Orchestrator (scripts + train_sd3.py)
3. Dataset Loader
– geneval, ocr, pickscore metadata readers & preprocessors
4. RL Engine (flow_grpo core)
– Environment & policy (qwenvl.py)
– Reward aggregator (rewards.py)
– Model update & EMA (ema.py)
– Stat tracking (stat_tracking.py)
5. Diffusion Pipeline (patched diffusers)
– sd3_sde_with_logprob, sd3_pipeline_with_logprob
6. Reward Scorers (local)
– aesthetic_scorer, imagereward_scorer, pickscore_scorer, ocr.py
7. External Reward Servers
– GenEval reward-server (remote endpoint)
– DeQA or other remote services
8. Compute Infrastructure
– Single-node/Multi-node cluster
– Deepspeed/Accelerate
9. Artifact Storage
– Model checkpoints (Hugging Face), logs, metrics
1. Orchestrator → Configuration Service (load hyperparameters)
2. Orchestrator → Dataset Loader (fetch prompts & metadata)
3. RL Engine ↔ Diffusion Pipeline (generate candidate images & compute log-probs)
4. RL Engine → Reward Scorers (send images)
• Local scorers (in-process function calls)
• External servers (HTTP/gRPC calls)
5. Reward Scorers → RL Engine (return reward values)
6. RL Engine → Compute Infrastructure (gradient steps via Deepspeed/Accelerate)
7. RL Engine → Artifact Storage (save checkpoints, logs)
8. Stat Tracking → Artifact Storage (metrics dashboards)
c. Architectural patterns & principles:
• Modular, plugin-based reward interface – new reward models can be “plugged in.”
• Clean separation between orchestration (scripts), configuration, data loading, core RL logic, and scoring.
• Use of distributed training frameworks (Deepspeed/Accelerate) abstracts hardware.
• Patches to standard libraries (diffusers) to expose additional signals (log-probs).
d. Key technologies/frameworks:
• Python 3.10, Conda environments
• Hugging Face diffusers + custom patches
• ddpo-pytorch design for RL training loops
• Deepspeed/Accelerate for multi-GPU/multi-node
• PaddleOCR for OCR reward
• External reward-server (GenEval)
– Blue rectangles: core components (config, orchestrator, RL engine)
– Green ovals: data sources (dataset loaders)
– Orange hexagons: reward scorers (local vs. remote)
– Gray clouds: external services (GenEval server)