How gen-verse/scoreflow works
Project type and purpose
• ScoreFlow is a research-oriented machine-learning workflow pipeline for optimizing multi-agent LLM (Large Language Model) workflows via score-based preference optimization (Score-DPO).
• It’s not a web app or microservices system—it’s a CLI-driven ML experiment framework comprising data ingestion, generation, evaluation, and optimization stages.
File structure
• Top-level CLI scripts:
• generate.py – run generation (inference or optimization prep)
• evaluate.py – compute scores/metrics on generated outputs
• optimize.py – perform Score-DPO training (via accelerate)
• get_scores.py – utility to collate or export results
• Configuration:
• config/config1.yaml – hardware & runtime settings (GPU, batch sizes)
• config/config2.yaml – API keys (e.g., OpenAI, Huggingface)
• Core library (ScoreFlow/):
• DPOconfig.py – load and manage hyperparameters
• DPOtrainer.py – define the Score-DPO optimization loop
• params.py – experiment default parameters
• benchmark/ – abstractions for each dataset benchmark (GSM8K, DROP, etc.)
• scripts/ – per-benchmark modules (conditions, extraction, judger, op_prompt, operator, operator_an) encapsulating dataset-specific logic
• Supporting files:
• requirements.txt – Python dependencies (PyTorch, Transformers, accelerate, MetaGPT)
• figures/ – pipeline.png, table.png for documentation
Extract insights from the README
• Pipeline stages: Data preparation → generation → evaluation → optimization, repeated iteratively over epochs.
• Core components: ScoreFlow library, external LLM APIs, accelerate for distributed training, MetaGPT for agent orchestration.
• Benchmarks: Six tasks (HumanEval, MBPP, MATH, GSM8K, DROP, HotpotQA) with corresponding dataset-specific script modules.
Main components
a. Data Layer
• Raw datasets (downloaded into project root)
b. Config Layer
• config1.yaml, config2.yaml
c. Core Library (ScoreFlow/)
• DPOconfig, DPOtrainer, params
d. Benchmark Modules
• Shared benchmark API (benchmark/*.py)
• Dataset-specific logic (scripts/<Benchmark>/*)
e. CLI Orchestrators
• generate.py
• evaluate.py
• optimize.py
• get_scores.py
f. External Services & Dependencies
• LLM APIs (OpenAI, BloomRPC, etc.)
• MetaGPT agent framework
• accelerate (distributed training)
• PyTorch, Transformers
Relationships and interactions
• generate.py reads config, loads benchmark scripts, calls MetaGPT + LLM APIs to produce candidate workflows/responses.
• evaluate.py ingests generated outputs + ground‐truth, calls benchmark.judger modules to score results.
• optimize.py uses DPOtrainer (Score-DPO) to compute gradients and update the policy; runs under accelerate.
• All CLI scripts consume Data and Config and depend on Core Library; Core Library invokes Benchmark Modules.
• Results feed back into the next generate→evaluate→optimize cycle (iterative optimization loop).
Architectural patterns and design principles
• Modular layering: clear separation of concerns between configuration, core logic, dataset-specific logic, and orchestration scripts.
• Pipeline pattern: sequential stages (Generate → Evaluate → Optimize) forming an iterative loop.
• Extensibility: adding a new benchmark only requires implementing scripts/<NewBenchmark>/*.py and adding to benchmark API.