# cjhirwa/agent-evaluation-harness architecture

> Architecture diagram of the GitHub repository cjhirwa/agent-evaluation-harness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cjhirwa/agent-evaluation-harness
- Repository: https://github.com/cjhirwa/agent-evaluation-harness

## Overview

This repository is an LLM-agent benchmark harness for simulated online-store tasks. A task definition and run configuration drive an episode: the runner builds a fresh sandbox world, an agent design consults a model provider and dispatches tool calls, then deterministic checks and scoring evaluate the answer and resulting state. Reports aggregate episode results. The README documents the major capabilities; some internal links below rely on its workflow or the SOURCE INDEX because the corresponding source was not excerpted.

Main entry points: [`aeh/runner.py`](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/runner.py) (Episode runner), [`aeh/cli.py`](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/cli.py) (CLI), [`aeh/providers/base.py`](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/base.py) (Provider interface), [`aeh/sandbox/world.py`](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/sandbox/world.py) (Sandbox world)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_scenario["Tasks &amp; world"]
  node_task_defs["Task YAML"]
  node_task_loader["Task loader<br/>[tasks.py]"]
  node_fixtures["Seed fixtures<br/>[fixtures.py]"]
  node_world[("Sandbox world<br/>[world.py]")]
  node_python_exec["Python sandbox<br/>[python_exec.py]"]
end

subgraph group_execution["Episode execution"]
  node_cli["CLI<br/>[cli.py]"]
  node_run_config["Run configs"]
  node_runner["Episode runner<br/>[runner.py]"]
  node_tools["Tool dispatcher<br/>[tools.py]"]
end

subgraph group_models["Agents &amp; providers"]
  node_agent_designs["Agent designs<br/>[agents.py]"]
  node_provider_iface["Provider interface<br/>[base.py]"]
  node_openai["OpenAI adapter<br/>[openai_provider.py]"]
  node_ollama["Ollama adapter<br/>[ollama_provider.py]"]
  node_scripted["Scripted provider<br/>[scripted.py]"]
end

subgraph group_evaluation["Evaluation &amp; results"]
  node_checks["Deterministic checks<br/>[checks.py]"]
  node_scoring["Episode scoring<br/>[scoring.py]"]
  node_reporting["Reports &amp; charts<br/>[report.py]"]
  node_episode_results["Episode results"]
end

node_user(("Benchmark user"))
node_model_services{{"Model services"}}

node_user -->|"runs commands"| node_cli
node_cli -->|"loads config"| node_run_config
node_cli -->|"lists tasks"| node_task_loader
node_cli -->|"starts run"| node_runner
node_cli -->|"requests report"| node_reporting
node_task_loader -->|"loads YAML"| node_task_defs
node_runner -->|"loads tasks"| node_task_loader
node_runner -->|"builds episode"| node_world
node_world -->|"seeds state"| node_fixtures
node_runner -->|"runs agent"| node_agent_designs
node_agent_designs -->|"requests completion"| node_provider_iface
node_agent_designs -->|"dispatches calls"| node_tools
node_tools -->|"reads and mutates"| node_world
node_tools -->|"executes code"| node_python_exec
node_provider_iface -.->|"implemented by"| node_openai
node_provider_iface -->|"implemented by"| node_ollama
node_provider_iface -->|"implemented by"| node_scripted
node_openai -.->|"calls API"| node_model_services
node_ollama -->|"calls Ollama"| node_model_services
node_runner -->|"checks episode"| node_checks
node_checks -->|"reads final state"| node_world
node_runner -->|"scores episode"| node_scoring
node_runner -->|"writes records"| node_episode_results
node_reporting -.->|"aggregates records"| node_episode_results
node_reporting -->|"uses score labels"| node_scoring

click node_cli "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/cli.py"
click node_run_config "https://github.com/cjhirwa/agent-evaluation-harness/tree/main/configs"
click node_task_defs "https://github.com/cjhirwa/agent-evaluation-harness/tree/main/tasks"
click node_task_loader "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/tasks.py"
click node_runner "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/runner.py"
click node_fixtures "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/sandbox/fixtures.py"
click node_world "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/sandbox/world.py"
click node_tools "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/tools.py"
click node_python_exec "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/sandbox/python_exec.py"
click node_agent_designs "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/agents.py"
click node_provider_iface "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/base.py"
click node_openai "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/openai_provider.py"
click node_ollama "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/ollama_provider.py"
click node_scripted "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/scripted.py"
click node_checks "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/checks.py"
click node_scoring "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/scoring.py"
click node_reporting "https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/report.py"
click node_episode_results "https://github.com/cjhirwa/agent-evaluation-harness/tree/main/results"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_task_defs,node_task_loader,node_fixtures,node_world,node_python_exec,node_user toneBlue
class node_cli,node_run_config,node_runner,node_tools toneAmber
class node_agent_designs,node_provider_iface,node_openai,node_ollama,node_scripted toneMint
class node_checks,node_scoring,node_reporting,node_episode_results toneRose
class node_model_services toneIndigo
```

## Components

### Tasks & world

- [Task YAML](https://github.com/cjhirwa/agent-evaluation-harness/tree/main/tasks): `tasks`
- [Task loader](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/tasks.py): `aeh/tasks.py`
- [Seed fixtures](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/sandbox/fixtures.py): `aeh/sandbox/fixtures.py`
- [Sandbox world](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/sandbox/world.py): `aeh/sandbox/world.py`
- [Python sandbox](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/sandbox/python_exec.py): `aeh/sandbox/python_exec.py`

### Episode execution

- [CLI](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/cli.py): `aeh/cli.py`
- [Run configs](https://github.com/cjhirwa/agent-evaluation-harness/tree/main/configs): `configs`
- [Episode runner](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/runner.py): `aeh/runner.py`
- [Tool dispatcher](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/tools.py): `aeh/tools.py`

### Agents & providers

- [Agent designs](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/agents.py): `aeh/agents.py`
- [Provider interface](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/base.py): `aeh/providers/base.py`
- [OpenAI adapter](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/openai_provider.py): `aeh/providers/openai_provider.py`
- [Ollama adapter](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/ollama_provider.py): `aeh/providers/ollama_provider.py`
- [Scripted provider](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/providers/scripted.py): `aeh/providers/scripted.py`

### Evaluation & results

- [Deterministic checks](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/checks.py): `aeh/checks.py`
- [Episode scoring](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/scoring.py): `aeh/scoring.py`
- [Reports & charts](https://github.com/cjhirwa/agent-evaluation-harness/blob/main/aeh/report.py): `aeh/report.py`
- [Episode results](https://github.com/cjhirwa/agent-evaluation-harness/tree/main/results): `results`

### Other components

- **Benchmark user**
- **Model services**

## Connections

- Benchmark user → CLI: runs commands (evidence: `README.md`)
- CLI → Run configs: loads config (evidence: `README.md`)
- CLI → Task loader: lists tasks (evidence: `aeh/cli.py`)
- CLI → Episode runner: starts run (evidence: `aeh/cli.py`)
- CLI → Reports & charts: requests report (evidence: `aeh/cli.py`)
- Task loader → Task YAML: loads YAML (evidence: `aeh/tasks.py`)
- Episode runner → Task loader: loads tasks (evidence: `aeh/runner.py`)
- Episode runner → Sandbox world: builds episode (evidence: `aeh/runner.py`)
- Sandbox world → Seed fixtures: seeds state (evidence: `aeh/sandbox/world.py`)
- Episode runner → Agent designs: runs agent (evidence: `aeh/runner.py`)
- Agent designs → Provider interface: requests completion (evidence: `aeh/agents.py`)
- Agent designs → Tool dispatcher: dispatches calls (evidence: `aeh/agents.py`)
- Tool dispatcher → Sandbox world: reads and mutates (evidence: `aeh/tools.py`)
- Tool dispatcher → Python sandbox: executes code (evidence: `aeh/tools.py`)
- Provider interface → OpenAI adapter: implemented by (evidence: `README.md`)
- Provider interface → Ollama adapter: implemented by (evidence: `aeh/providers/ollama_provider.py`)
- Provider interface → Scripted provider: implemented by (evidence: `aeh/providers/scripted.py`)
- OpenAI adapter → Model services: calls API (evidence: `README.md`)
- Ollama adapter → Model services: calls Ollama (evidence: `aeh/providers/ollama_provider.py`)
- Episode runner → Deterministic checks: checks episode (evidence: `aeh/runner.py`)
- Deterministic checks → Sandbox world: reads final state (evidence: `aeh/checks.py`)
- Episode runner → Episode scoring: scores episode (evidence: `aeh/runner.py`)
- Episode runner → Episode results: writes records (evidence: `aeh/runner.py`)
- Reports & charts → Episode results: aggregates records (evidence: `README.md`)
- Reports & charts → Episode scoring: uses score labels (evidence: `aeh/report.py`)
