# darthvader19/agent_eval architecture

> Architecture diagram of the GitHub repository darthvader19/agent_eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darthvader19/agent_eval
- Repository: https://github.com/darthvader19/agent_eval

## Overview

This is an agent evaluation and benchmarking framework. Its main workflow loads benchmark tasks and experiment settings, runs selected agent architectures under shared budgets and tools, grades answers, then persists repeat-level results for analysis. The map includes documented reflection and optional framework comparisons. Some components are represented by directory paths because their implementation was not sampled; links to them are included only where the README or SOURCE INDEX supports the relationship.

Main entry points: [`harness/runner.py`](https://github.com/darthvader19/agent_eval/blob/main/harness/runner.py) (Evaluation runner), [`tools/toolbox.py`](https://github.com/darthvader19/agent_eval/blob/main/tools/toolbox.py) (Shared tool interface), [`agents/plan_execute_agent.py`](https://github.com/darthvader19/agent_eval/blob/main/agents/plan_execute_agent.py) (Plan and execute), [`agents/langgraph_agent.py`](https://github.com/darthvader19/agent_eval/blob/main/agents/langgraph_agent.py) (LangGraph agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Evaluation workflow"]
  node_runner["Evaluation runner<br/>[runner.py]"]
  node_tasks["Task suites"]
  node_config["Experiment settings"]
end

subgraph group_agents["Agent architectures"]
  node_base["Agent contract<br/>[base.py]"]
  node_react["ReAct agent<br/>[react_agent.py]"]
  node_plan["Plan and execute"]
  node_langgraph["LangGraph agent<br/>[langgraph_agent.py]"]
  node_direct["Direct baseline<br/>[direct_baseline.py]"]
  node_frameworks["Framework adapters"]
  node_runtime["Step execution<br/>[runtime.py]"]
end

subgraph group_tools["Shared tools"]
  node_toolbox["Shared tool interface<br/>[toolbox.py]"]
  node_calculator["Calculator<br/>[calculator.py]"]
  node_python["Python sandbox<br/>[python_sandbox.py]"]
  node_search["Snapshot search<br/>[search.py]"]
  node_retriever["Local retrieval<br/>[retriever.py]"]
  node_faults["Fault injection<br/>[fault_injection.py]"]
  node_corpus["Benchmark corpus"]
end

subgraph group_scoring["Scoring and analysis"]
  node_judge["Answer grading<br/>[judge.py]"]
  node_reflection["Paired reflection<br/>[reflection.py]"]
  node_metrics["Run metrics<br/>[metrics.py]"]
  node_stats["Statistical summaries<br/>[stats.py]"]
  node_analysis["Result analysis"]
end

subgraph group_integrations["Models and persistence"]
  node_model["Model client<br/>[model.py]"]
  node_provider["Chat completions API"]
  node_results[("Run results")]
end

node_user(("Evaluator"))

node_user -->|"starts run"| node_runner
node_runner -->|"loads tasks"| node_tasks
node_runner -->|"uses settings"| node_config
node_runner -->|"runs agent"| node_react
node_runner -->|"runs agent"| node_plan
node_runner -->|"runs agent"| node_langgraph
node_runner -->|"runs agent"| node_direct
node_runner -.->|"runs adapters"| node_frameworks
node_react -.->|"implements"| node_base
node_plan -->|"executes steps"| node_runtime
node_langgraph -->|"executes steps"| node_runtime
node_runtime -->|"invokes tools"| node_toolbox
node_plan -->|"uses tools"| node_toolbox
node_langgraph -->|"uses tools"| node_toolbox
node_frameworks -->|"shares interface"| node_toolbox
node_toolbox -->|"dispatches to"| node_calculator
node_toolbox -->|"dispatches to"| node_python
node_toolbox -->|"dispatches to"| node_search
node_toolbox -->|"dispatches to"| node_retriever
node_toolbox -->|"applies faults"| node_faults
node_search -->|"reads snapshots"| node_corpus
node_retriever -->|"reads documents"| node_corpus
node_runner -->|"grades answers"| node_judge
node_runner -->|"runs reflection"| node_reflection
node_runner -->|"aggregates metrics"| node_metrics
node_metrics -->|"uses statistics"| node_stats
node_analysis -->|"analyzes runs"| node_results
node_runner -->|"persists results"| node_results
node_base -->|"creates client"| node_model
node_judge -->|"uses for LLM grading"| node_model
node_model -->|"sends completions"| node_provider

click node_runner "https://github.com/darthvader19/agent_eval/blob/main/harness/runner.py"
click node_tasks "https://github.com/darthvader19/agent_eval/tree/main/benchmarks"
click node_config "https://github.com/darthvader19/agent_eval/tree/main/configs/experiments"
click node_base "https://github.com/darthvader19/agent_eval/blob/main/agents/base.py"
click node_react "https://github.com/darthvader19/agent_eval/blob/main/agents/react_agent.py"
click node_plan "https://github.com/darthvader19/agent_eval/blob/main/agents/plan_execute_agent.py"
click node_langgraph "https://github.com/darthvader19/agent_eval/blob/main/agents/langgraph_agent.py"
click node_direct "https://github.com/darthvader19/agent_eval/blob/main/agents/direct_baseline.py"
click node_frameworks "https://github.com/darthvader19/agent_eval/tree/main/frameworks"
click node_runtime "https://github.com/darthvader19/agent_eval/blob/main/agents/runtime.py"
click node_toolbox "https://github.com/darthvader19/agent_eval/blob/main/tools/toolbox.py"
click node_calculator "https://github.com/darthvader19/agent_eval/blob/main/tools/calculator.py"
click node_python "https://github.com/darthvader19/agent_eval/blob/main/tools/python_sandbox.py"
click node_search "https://github.com/darthvader19/agent_eval/blob/main/tools/search.py"
click node_retriever "https://github.com/darthvader19/agent_eval/blob/main/tools/retriever.py"
click node_faults "https://github.com/darthvader19/agent_eval/blob/main/tools/fault_injection.py"
click node_judge "https://github.com/darthvader19/agent_eval/blob/main/harness/judge.py"
click node_reflection "https://github.com/darthvader19/agent_eval/blob/main/harness/reflection.py"
click node_metrics "https://github.com/darthvader19/agent_eval/blob/main/harness/metrics.py"
click node_stats "https://github.com/darthvader19/agent_eval/blob/main/harness/stats.py"
click node_analysis "https://github.com/darthvader19/agent_eval/blob/main/analysis/aggregate_results.py"
click node_model "https://github.com/darthvader19/agent_eval/blob/main/harness/model.py"
click node_results "https://github.com/darthvader19/agent_eval/tree/main/results"
click node_corpus "https://github.com/darthvader19/agent_eval/tree/main/benchmarks/corpus"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_tasks,node_config toneBlue
class node_base,node_react,node_plan,node_langgraph,node_direct,node_frameworks,node_runtime toneAmber
class node_toolbox,node_calculator,node_python,node_search,node_retriever,node_faults,node_corpus toneMint
class node_judge,node_reflection,node_metrics,node_stats,node_analysis toneRose
class node_model,node_provider,node_results,node_user toneIndigo
```

## Components

### Evaluation workflow

- [Evaluation runner](https://github.com/darthvader19/agent_eval/blob/main/harness/runner.py): `harness/runner.py`
- [Task suites](https://github.com/darthvader19/agent_eval/tree/main/benchmarks): `benchmarks`
- [Experiment settings](https://github.com/darthvader19/agent_eval/tree/main/configs/experiments): `configs/experiments`

### Agent architectures

- [Agent contract](https://github.com/darthvader19/agent_eval/blob/main/agents/base.py): `agents/base.py`
- [ReAct agent](https://github.com/darthvader19/agent_eval/blob/main/agents/react_agent.py): `agents/react_agent.py`
- [Plan and execute](https://github.com/darthvader19/agent_eval/blob/main/agents/plan_execute_agent.py): `agents/plan_execute_agent.py`
- [LangGraph agent](https://github.com/darthvader19/agent_eval/blob/main/agents/langgraph_agent.py): `agents/langgraph_agent.py`
- [Direct baseline](https://github.com/darthvader19/agent_eval/blob/main/agents/direct_baseline.py): `agents/direct_baseline.py`
- [Framework adapters](https://github.com/darthvader19/agent_eval/tree/main/frameworks): `frameworks`
- [Step execution](https://github.com/darthvader19/agent_eval/blob/main/agents/runtime.py): `agents/runtime.py`

### Shared tools

- [Shared tool interface](https://github.com/darthvader19/agent_eval/blob/main/tools/toolbox.py): `tools/toolbox.py`
- [Calculator](https://github.com/darthvader19/agent_eval/blob/main/tools/calculator.py): `tools/calculator.py`
- [Python sandbox](https://github.com/darthvader19/agent_eval/blob/main/tools/python_sandbox.py): `tools/python_sandbox.py`
- [Snapshot search](https://github.com/darthvader19/agent_eval/blob/main/tools/search.py): `tools/search.py`
- [Local retrieval](https://github.com/darthvader19/agent_eval/blob/main/tools/retriever.py): `tools/retriever.py`
- [Fault injection](https://github.com/darthvader19/agent_eval/blob/main/tools/fault_injection.py): `tools/fault_injection.py`
- [Benchmark corpus](https://github.com/darthvader19/agent_eval/tree/main/benchmarks/corpus): `benchmarks/corpus`

### Scoring and analysis

- [Answer grading](https://github.com/darthvader19/agent_eval/blob/main/harness/judge.py): `harness/judge.py`
- [Paired reflection](https://github.com/darthvader19/agent_eval/blob/main/harness/reflection.py): `harness/reflection.py`
- [Run metrics](https://github.com/darthvader19/agent_eval/blob/main/harness/metrics.py): `harness/metrics.py`
- [Statistical summaries](https://github.com/darthvader19/agent_eval/blob/main/harness/stats.py): `harness/stats.py`
- [Result analysis](https://github.com/darthvader19/agent_eval/blob/main/analysis/aggregate_results.py): `analysis/aggregate_results.py`

### Models and persistence

- [Model client](https://github.com/darthvader19/agent_eval/blob/main/harness/model.py): `harness/model.py`
- **Chat completions API**
- [Run results](https://github.com/darthvader19/agent_eval/tree/main/results): `results`

### Other components

- **Evaluator**

## Connections

- Evaluator → Evaluation runner: starts run (evidence: `README.md`)
- Evaluation runner → Task suites: loads tasks (evidence: `README.md`)
- Evaluation runner → Experiment settings: uses settings (evidence: `README.md`)
- Evaluation runner → ReAct agent: runs agent (evidence: `harness/runner.py`)
- Evaluation runner → Plan and execute: runs agent (evidence: `harness/runner.py`)
- Evaluation runner → LangGraph agent: runs agent (evidence: `harness/runner.py`)
- Evaluation runner → Direct baseline: runs agent (evidence: `harness/runner.py`)
- Evaluation runner → Framework adapters: runs adapters (evidence: `README.md`)
- ReAct agent → Agent contract: implements (evidence: `agents/react_agent.py`)
- Plan and execute → Step execution: executes steps (evidence: `agents/plan_execute_agent.py`)
- LangGraph agent → Step execution: executes steps (evidence: `agents/langgraph_agent.py`)
- Step execution → Shared tool interface: invokes tools (evidence: `agents/runtime.py`)
- Plan and execute → Shared tool interface: uses tools (evidence: `agents/plan_execute_agent.py`)
- LangGraph agent → Shared tool interface: uses tools (evidence: `agents/langgraph_agent.py`)
- Framework adapters → Shared tool interface: shares interface (evidence: `frameworks/crewai_runner.py`)
- Shared tool interface → Calculator: dispatches to (evidence: `tools/toolbox.py`)
- Shared tool interface → Python sandbox: dispatches to (evidence: `tools/toolbox.py`)
- Shared tool interface → Snapshot search: dispatches to (evidence: `tools/toolbox.py`)
- Shared tool interface → Local retrieval: dispatches to (evidence: `tools/toolbox.py`)
- Shared tool interface → Fault injection: applies faults (evidence: `tools/toolbox.py`)
- Snapshot search → Benchmark corpus: reads snapshots (evidence: `tools/search.py`)
- Local retrieval → Benchmark corpus: reads documents (evidence: `tools/retriever.py`)
- Evaluation runner → Answer grading: grades answers (evidence: `harness/runner.py`)
- Evaluation runner → Paired reflection: runs reflection (evidence: `harness/runner.py`)
- Evaluation runner → Run metrics: aggregates metrics (evidence: `harness/runner.py`)
- Run metrics → Statistical summaries: uses statistics (evidence: `harness/metrics.py`)
- Result analysis → Run results: analyzes runs (evidence: `README.md`)
- Evaluation runner → Run results: persists results (evidence: `README.md`)
- Agent contract → Model client: creates client (evidence: `agents/base.py`)
- Answer grading → Model client: uses for LLM grading (evidence: `harness/judge.py`)
- Model client → Chat completions API: sends completions (evidence: `harness/model.py`)
