# justherozen/flowbench architecture

> Architecture diagram of the GitHub repository justherozen/flowbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/justherozen/flowbench
- Repository: https://github.com/justherozen/flowbench

## Overview

##### Project Type and Purpose
- FlowBench is an open-source research benchmark and evaluation framework (a command-line tool) for workflow-guided planning in LLM-based agents.
- It provides two pipelines—turn-level and session-level—for generating and evaluating LLM outputs against structured workflow knowledge.

##### High-Level File Structure Analysis
- Top-level:
- README.md – project description & usage
- requirements.txt – Python dependencies
- resources/ – static assets (flowchart overview image)
- script/*.sh – convenience wrappers for full pipelines
- session_level/ – Python modules for session-level simulation & metrics
- turn_level/ – Python modules for single-turn inference & metrics
- utils/ – common utilities (API wrappers, evaluation helpers, key config)

##### Core Components
a. Data Layer
- Input benchmark data (turn_data/, session_data/)
- knowledge.json (workflow knowledge in text/code/flowchart formats)
b. Processing Modules
- turn_level/turn_inference.py → generates single-turn responses via LLM
- turn_level/turn_metric_display.py → computes and displays turn-level metrics
- session_level/session_simulate.py → simulates multi-round interactions (modes: simulate, eval)
- session_level/session_metric_display.py → computes and displays session-level metrics
c. Utility Layer
- utils/llm.py & utils/request_openai.py → abstract OpenAI API calls
- utils/eval_utils.py → shared evaluation functions
- utils/utils.py → general helper routines
- utils/key.json → stores OPENAI_API_KEY
d. Orchestration Layer
- script/session_level.sh & script/turn_level.sh → shell entrypoints that wire data → modules
e. External Service
- OpenAI API (GPT-based models for generation, judging, simulation)
f. Output & Reporting
- OUTPUT_FOLDER for raw LLM responses
- EVAL_FOLDER for serialized evaluation results
- Excel or console reports (via metric_display scripts)

##### Relationships and Data Flow
- Final “metric_display” reads EVAL_FOLDER → prints reports or writes Excel

##### Architectural Patterns and Design Principles
- Layered pipeline: clear separation between data ingestion, processing, evaluation, and reporting
- Modularization: distinct sub-packages for turn-level vs. session-level concerns
- Abstraction of external dependencies (all API calls funnel through utils/request_openai.py)
- Reusability: shared evaluation utilities in utils/eval_utils.py

##### Relevant Technologies
- Python 3.x, shell scripting
- OpenAI GPT API
- JSON for knowledge and results serialization
- pandas/openpyxl (implied for Excel export in metric_display)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestration Layer
    subgraph "Orchestration Layer"
        direction TB
        TurnScript["turn_level.sh"]:::script
        SessionScript["session_level.sh"]:::script
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        TurnData["turn_data/"]:::data
        SessionData["session_data/"]:::data
        Knowledge["knowledge.json"]:::data
    end

    %% Processing Modules
    subgraph "Turn-Level Pipeline"
        direction TB
        Inference["turn_inference.py"]:::python
        TurnMetrics["turn_metric_display.py"]:::python
    end

    subgraph "Session-Level Pipeline"
        direction TB
        Simulate["session_simulate.py"]:::python
        SessionMetrics["session_metric_display.py"]:::python
    end

    %% Utility Layer
    subgraph "Utility Layer"
        direction TB
        LLM["llm.py"]:::python
        RequestAPI["request_openai.py"]:::python
        EvalUtils["eval_utils.py"]:::python
        Utils["utils.py"]:::python
        APIKey["key.json"]:::data
    end

    %% External Service
    OpenAI["OpenAI API"]:::external

    %% Output & Reporting
    subgraph "Output & Reporting"
        direction TB
        OutputRaw["OUTPUT_FOLDER"]:::data
        EvalResults["EVAL_FOLDER"]:::data
        Reports["Console/Excel Reports"]:::python
    end

    %% Project Files
    subgraph "Project Root Files"
        direction TB
        Readme["README.md"]:::data
        Requirements["requirements.txt"]:::data
        Resources["resources/"]:::data
    end

    %% Connections
    TurnScript -->|uses| TurnData
    TurnScript -->|uses| Knowledge
    TurnScript -->|runs| Inference

    SessionScript -->|uses| SessionData
    SessionScript -->|uses| Knowledge
    SessionScript -->|runs| Simulate

    Inference -->|writes| OutputRaw
    Simulate -->|writes| OutputRaw

    Inference -->|calls| RequestAPI
    Simulate -->|calls| RequestAPI
    RequestAPI -->|API call| OpenAI
    OpenAI -->|response| RequestAPI

    RequestAPI -->|provides| LLM
    APIKey -->|configures| RequestAPI

    Inference -->|passes responses| TurnMetrics
    TurnMetrics -->|writes| EvalResults
    EvalResults -->|reads| TurnMetrics
    TurnMetrics -->|outputs| Reports

    Simulate -->|passes sessions| SessionMetrics
    SessionMetrics -->|writes| EvalResults
    EvalResults -->|reads| SessionMetrics
    SessionMetrics -->|outputs| Reports

    EvalUtils -->|shared| TurnMetrics
    EvalUtils -->|shared| SessionMetrics
    Utils -->|helpers| Inference
    Utils -->|helpers| Simulate

    %% Click Events
    click TurnScript "https://github.com/justherozen/flowbench/blob/main/script/turn_level.sh"
    click SessionScript "https://github.com/justherozen/flowbench/blob/main/script/session_level.sh"
    click Inference "https://github.com/justherozen/flowbench/blob/main/turn_level/turn_inference.py"
    click TurnMetrics "https://github.com/justherozen/flowbench/blob/main/turn_level/turn_metric_display.py"
    click Simulate "https://github.com/justherozen/flowbench/blob/main/session_level/session_simulate.py"
    click SessionMetrics "https://github.com/justherozen/flowbench/blob/main/session_level/session_metric_display.py"
    click RequestAPI "https://github.com/justherozen/flowbench/blob/main/utils/request_openai.py"
    click LLM "https://github.com/justherozen/flowbench/blob/main/utils/llm.py"
    click EvalUtils "https://github.com/justherozen/flowbench/blob/main/utils/eval_utils.py"
    click Utils "https://github.com/justherozen/flowbench/blob/main/utils/utils.py"
    click APIKey "https://github.com/justherozen/flowbench/blob/main/utils/key.json"
    click Readme "https://github.com/justherozen/flowbench/blob/main/README.md"
    click Requirements "https://github.com/justherozen/flowbench/blob/main/requirements.txt"
    click Resources "https://github.com/justherozen/flowbench/tree/main/resources/"

    %% Styles
    classDef python fill:#AED6F1,stroke:#1B4F72,stroke-width:1px
    classDef data fill:#ABEBC6,stroke:#145A32,stroke-width:1px
    classDef script fill:#FCF3CF,stroke:#7D6608,stroke-width:1px
    classDef external fill:#F5B7B1,stroke:#641E16,stroke-width:1px
```
