# gen-verse/scoreflow architecture

> Architecture diagram of the GitHub repository gen-verse/scoreflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gen-verse/scoreflow
- Repository: https://github.com/gen-verse/scoreflow

## Overview

##### Project type and purpose
- ScoreFlow is a research-oriented machine-learning workflow pipeline for optimizing multi-agent LLM (Large Language Model) workflows via score-based preference optimization (Score-DPO).
- It’s not a web app or microservices system—it’s a CLI-driven ML experiment framework comprising data ingestion, generation, evaluation, and optimization stages.

##### File structure
- Top-level CLI scripts:
- generate.py – run generation (inference or optimization prep)
- evaluate.py – compute scores/metrics on generated outputs
- optimize.py – perform Score-DPO training (via accelerate)
- get_scores.py – utility to collate or export results
- Configuration:
- config/config1.yaml – hardware & runtime settings (GPU, batch sizes)
- config/config2.yaml – API keys (e.g., OpenAI, Huggingface)
- Core library (ScoreFlow/):
- DPOconfig.py – load and manage hyperparameters
- DPOtrainer.py – define the Score-DPO optimization loop
- params.py – experiment default parameters
- benchmark/ – abstractions for each dataset benchmark (GSM8K, DROP, etc.)
- scripts/ – per-benchmark modules (conditions, extraction, judger, op_prompt, operator, operator_an) encapsulating dataset-specific logic
- Supporting files:
- requirements.txt – Python dependencies (PyTorch, Transformers, accelerate, MetaGPT)
- figures/ – pipeline.png, table.png for documentation

##### Extract insights from the README
- Pipeline stages: Data preparation → generation → evaluation → optimization, repeated iteratively over epochs.
- Core components: ScoreFlow library, external LLM APIs, accelerate for distributed training, MetaGPT for agent orchestration.
- Benchmarks: Six tasks (HumanEval, MBPP, MATH, GSM8K, DROP, HotpotQA) with corresponding dataset-specific script modules.

##### Main components
a. Data Layer
- Raw datasets (downloaded into project root)
b. Config Layer
- config1.yaml, config2.yaml
c. Core Library (ScoreFlow/)
- DPOconfig, DPOtrainer, params
d. Benchmark Modules
- Shared benchmark API (benchmark/*.py)
- Dataset-specific logic (scripts/<Benchmark>/*)
e. CLI Orchestrators
- generate.py
- evaluate.py
- optimize.py
- get_scores.py
f. External Services & Dependencies
- LLM APIs (OpenAI, BloomRPC, etc.)
- MetaGPT agent framework
- accelerate (distributed training)
- PyTorch, Transformers

##### Relationships and interactions
- generate.py reads config, loads benchmark scripts, calls MetaGPT + LLM APIs to produce candidate workflows/responses.
- evaluate.py ingests generated outputs + ground‐truth, calls benchmark.judger modules to score results.
- optimize.py uses DPOtrainer (Score-DPO) to compute gradients and update the policy; runs under accelerate.
- All CLI scripts consume Data and Config and depend on Core Library; Core Library invokes Benchmark Modules.
- Results feed back into the next generate→evaluate→optimize cycle (iterative optimization loop).

##### Architectural patterns and design principles
- Modular layering: clear separation of concerns between configuration, core logic, dataset-specific logic, and orchestration scripts.
- Pipeline pattern: sequential stages (Generate → Evaluate → Optimize) forming an iterative loop.
- Extensibility: adding a new benchmark only requires implementing scripts/<NewBenchmark>/*.py and adding to benchmark API.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Config"
        classDef blue fill:#D6E9F8,stroke:#4D89F9
        Data["Raw Datasets"]:::blue
        Config1["config1.yaml"]:::blue
        Config2["config2.yaml"]:::blue
    end

    subgraph "Core Library"
        classDef green fill:#DFF2BF,stroke:#4F8A10
        DPOconfig["DPOconfig.py"]:::green
        DPOtrainer["DPOtrainer.py"]:::green
        Params["params.py"]:::green
        BenchmarkAPI["benchmark.py"]:::green
    end

    subgraph "Benchmark Modules"
        classDef green fill:#DFF2BF,stroke:#4F8A10
        GSM8Kmod["gsm8k.py"]:::green
        DROPmod["drop.py"]:::green
        Hotpotmod["hotpotqa.py"]:::green
        HumanEvalmod["humaneval.py"]:::green
        MATHmod["math.py"]:::green
        MBPPmod["mbpp.py"]:::green
    end

    subgraph "Dataset-Specific Scripts"
        classDef green fill:#DFF2BF,stroke:#4F8A10
        GSM8Kscripts["GSM8K scripts"]:::green
        DROPscripts["DROP scripts"]:::green
        Hotpotscripts["HotpotQA scripts"]:::green
        HumanEvalscripts["HumanEval scripts"]:::green
        MATHscripts["MATH scripts"]:::green
        MBPPscripts["MBPP scripts"]:::green
    end

    subgraph "CLI Orchestrators"
        classDef orange fill:#FEEFB3,stroke:#9F6000
        Gen["generate.py"]:::orange
        Eval["evaluate.py"]:::orange
        Opt["optimize.py"]:::orange
        Scores["get_scores.py"]:::orange
    end

    subgraph "External Services & Dependencies"
        classDef purple fill:#E1BEE7,stroke:#7B1FA2
        LLM["LLM APIs (OpenAI, Huggingface)"]:::purple
        MetaGPT["MetaGPT agents"]:::purple
        Accelerate["accelerate"]:::purple
        Torch["PyTorch, Transformers"]:::purple
    end

    Data -->|feeds into| Gen
    Config1 -->|loads| Gen
    Config2 -->|loads| Gen

    Gen -->|calls| MetaGPT
    Gen -->|requests| LLM
    Gen -->|uses| BenchmarkAPI

    Gen -->|outputs to| Eval
    Eval -->|scoring with| BenchmarkAPI
    Eval -->|references| Data

    Eval -->|passes results to| Opt
    Opt -->|trains via| DPOtrainer
    Opt -->|uses| Accelerate
    Opt -->|updates| Params

    Opt -->|iterates back to| Gen

    Scores -->|aggregates from| Eval

    Gen -->|depends on| DPOconfig
    Gen -->|depends on| Params
    Gen -->|depends on| BenchmarkAPI

    Eval -->|depends on| DPOconfig
    Eval -->|depends on| Params
    Eval -->|depends on| BenchmarkAPI

    Opt -->|depends on| DPOconfig
    Opt -->|depends on| Params
    Opt -->|depends on| BenchmarkAPI

    Scores -->|depends on| DPOconfig
    Scores -->|depends on| Params
    Scores -->|depends on| BenchmarkAPI

    BenchmarkAPI --> GSM8Kmod
    BenchmarkAPI --> DROPmod
    BenchmarkAPI --> Hotpotmod
    BenchmarkAPI --> HumanEvalmod
    BenchmarkAPI --> MATHmod
    BenchmarkAPI --> MBPPmod

    GSM8Kmod --> GSM8Kscripts
    DROPmod --> DROPscripts
    Hotpotmod --> Hotpotscripts
    HumanEvalmod --> HumanEvalscripts
    MATHmod --> MATHscripts
    MBPPmod --> MBPPscripts

    Torch --> DPOtrainer
    DPOconfig --> DPOtrainer
    Params --> DPOtrainer

    click Config1 "https://github.com/gen-verse/scoreflow/blob/main/config/config1.yaml"
    click Config2 "https://github.com/gen-verse/scoreflow/blob/main/config/config2.yaml"
    click DPOconfig "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/DPOconfig.py"
    click DPOtrainer "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/DPOtrainer.py"
    click Params "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/params.py"
    click BenchmarkAPI "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/benchmark/benchmark.py"
    click GSM8Kmod "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/benchmark/gsm8k.py"
    click DROPmod "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/benchmark/drop.py"
    click Hotpotmod "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/benchmark/hotpotqa.py"
    click HumanEvalmod "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/benchmark/humaneval.py"
    click MATHmod "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/benchmark/math.py"
    click MBPPmod "https://github.com/gen-verse/scoreflow/blob/main/ScoreFlow/benchmark/mbpp.py"
    click GSM8Kscripts "https://github.com/gen-verse/scoreflow/tree/main/ScoreFlow/scripts/GSM8K/"
    click DROPscripts "https://github.com/gen-verse/scoreflow/tree/main/ScoreFlow/scripts/DROP/"
    click Hotpotscripts "https://github.com/gen-verse/scoreflow/tree/main/ScoreFlow/scripts/HotpotQA/"
    click HumanEvalscripts "https://github.com/gen-verse/scoreflow/tree/main/ScoreFlow/scripts/HumanEval/"
    click MATHscripts "https://github.com/gen-verse/scoreflow/tree/main/ScoreFlow/scripts/MATH/"
    click MBPPscripts "https://github.com/gen-verse/scoreflow/tree/main/ScoreFlow/scripts/MBPP/"
    click Gen "https://github.com/gen-verse/scoreflow/blob/main/generate.py"
    click Eval "https://github.com/gen-verse/scoreflow/blob/main/evaluate.py"
    click Opt "https://github.com/gen-verse/scoreflow/blob/main/optimize.py"
    click Scores "https://github.com/gen-verse/scoreflow/blob/main/get_scores.py"
    click requirements "https://github.com/gen-verse/scoreflow/blob/main/requirements.txt"
    click README "https://github.com/gen-verse/scoreflow/blob/main/README.md"
    
    class Data,Config1,Config2 blue
    class DPOconfig,DPOtrainer,Params,BenchmarkAPI,GSM8Kmod,DROPmod,Hotpotmod,HumanEvalmod,MATHmod,MBPPmod,GSM8Kscripts,DROPscripts,Hotpotscripts,HumanEvalscripts,MATHscripts,MBPPscripts green
    class Gen,Eval,Opt,Scores orange
    class LLM,MetaGPT,Accelerate,Torch purple
```
