How justherozen/flowbench works
Project Type and Purpose
• FlowBench is an open-source research benchmark and evaluation framework (a command-line tool) for workflow-guided planning in LLM-based agents.
• It provides two pipelines—turn-level and session-level—for generating and evaluating LLM outputs against structured workflow knowledge.
High-Level File Structure Analysis
• Top-level:
• README.md – project description & usage
• requirements.txt – Python dependencies
• resources/ – static assets (flowchart overview image)
• script/*.sh – convenience wrappers for full pipelines
• session_level/ – Python modules for session-level simulation & metrics
• turn_level/ – Python modules for single-turn inference & metrics
• utils/ – common utilities (API wrappers, evaluation helpers, key config)
Core Components
a. Data Layer
• Input benchmark data (turn_data/, session_data/)
• knowledge.json (workflow knowledge in text/code/flowchart formats)
b. Processing Modules
• turn_level/turn_inference.py → generates single-turn responses via LLM
• turn_level/turn_metric_display.py → computes and displays turn-level metrics
• session_level/session_simulate.py → simulates multi-round interactions (modes: simulate, eval)
• session_level/session_metric_display.py → computes and displays session-level metrics
c. Utility Layer
• utils/llm.py & utils/request_openai.py → abstract OpenAI API calls
• utils/eval_utils.py → shared evaluation functions
• utils/utils.py → general helper routines
• utils/key.json → stores OPENAI_API_KEY
d. Orchestration Layer
• script/session_level.sh & script/turn_level.sh → shell entrypoints that wire data → modules
e. External Service
• OpenAI API (GPT-based models for generation, judging, simulation)
f. Output & Reporting
• OUTPUT_FOLDER for raw LLM responses
• EVAL_FOLDER for serialized evaluation results
• Excel or console reports (via metric_display scripts)
Relationships and Data Flow
• Final “metric_display” reads EVAL_FOLDER → prints reports or writes Excel
Architectural Patterns and Design Principles
• Layered pipeline: clear separation between data ingestion, processing, evaluation, and reporting
• Modularization: distinct sub-packages for turn-level vs. session-level concerns
• Abstraction of external dependencies (all API calls funnel through utils/request_openai.py)
• Reusability: shared evaluation utilities in utils/eval_utils.py
Relevant Technologies
• Python 3.x, shell scripting
• OpenAI GPT API
• JSON for knowledge and results serialization
• pandas/openpyxl (implied for Excel export in metric_display)