# apodexai/agentharness architecture

> Architecture diagram of the GitHub repository apodexai/agentharness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apodexai/agentharness
- Repository: https://github.com/apodexai/agentharness

## Overview

1. **What this repo is**
   AgentHarness is a Python 3.12 evaluation harness for running deep-research agents against public benchmarks. It is not a product app; it is an execution framework that loads a benchmark task, runs an agent workflow, captures events, and scores the output with benchmark-specific judges.

2. **Top-level execution flow**
   The main path is: benchmark dataset task → benchmark runner → subprocess-per-question execution → selected workflow/pipeline → agent loop with model calls and tools → event/log capture → judge/accuracy aggregation. The subprocess boundary is important: each question is isolated for reproducibility, hang recovery, and independent reruns.

3. **Benchmarking subsystem**
   The `benchmarks/` package is the orchestration layer for experiments. It contains dataset/family definitions, task generation, answer extraction, judge logic, and runners such as `run_subprocess.py` and `check_progress.py`. This is where benchmark-specific inputs and scoring live, separate from the agent runtime itself.

4. **Workflow layer**
   The `workflows/react_base/` package defines the canonical ReAct-style agent configuration used in the paper. It includes the main agent node, prompt/spec/profile definitions, and workflow-specific observers and compactors. Profiles like `default.yaml` and `keep5.yaml` alter runtime behavior without changing the core engine.

5. **Core runtime engine**
   The `agent_harness/core/runtime/` tree is the central execution engine. It contains the agent loop, tool-call parsing, tool execution, context-budget handling, compaction, pause checks, event bus, DAG utilities, registries, and resource/permission management. This is the boundary between “workflow description” and “actual runtime behavior.”

6. **Model and LLM integration**
   LLM access is abstracted through `agent_harness/core/llm.py`, `infra/llm_adapter.py`, and provider clients for OpenAI- and Anthropic-compatible endpoints. In practice, the README’s default path points `OPENAI_BASE_URL`/`OPENAI_API_KEY`/`OPENAI_MODEL` at a model server such as SGLang. The runtime is designed around OpenAI-style chat/tool-call interfaces, with model-profile logic controlling parsing and context behavior.

7. **Tooling and external services**
   Tools are first-class, but are isolated in `plugins/tools/` and permissioned through runtime resource managers. The concrete external integrations are web search, web fetch, and Python code execution, backed by services like Serper, Jina, and E2B. This creates a clear boundary: the agent can request tools, but execution is mediated by the runtime and permissions layer.

8. **Observers and control signals**
   Observers appear in both `agent_harness/components/observers/` and `workflows/react_base/observers/`. They are side-channel processors that watch runtime events and enforce or react to conditions such as budget limits, context-size guards, rollback on duplicate/empty/refusal cases, tool-call normalization, trajectory capture, and SSE/console output. They do not own the agent loop; they modify or annotate it.

9. **State and event persistence**
   The state layer is intentionally lightweight: `agent_harness/state/event_store/sqlite.py` provides a local event store for recording run traces. This supports debugging, replay, and post-run analysis without coupling the runtime to a larger database architecture.

10. **Scheduling and process management**
   The `agent_harness/scheduling/` package manages workflow loading, pipeline registration, and process orchestration. Combined with the benchmark runner, this is the experiment-control plane: it resolves which pipeline/workflow to run, launches work, and coordinates multi-run evaluation.

11. **Data model and shared types**
   The `models/` and `core/` packages define the shared contracts: agent definitions, tasks, pipeline specs, node context, events, messages, errors, and protocol interfaces. These types are the internal API between benchmarks, workflows, runtime, and observers, and are key to understanding the repo boundary structure.

12. **Operational dependencies**
   The repo assumes several external runtime inputs: benchmark datasets on disk, an OpenAI-compatible model endpoint, and optional web/code service credentials from `.env`. CI exists via GitHub Actions, and benchmark scripts in `scripts/` provide canned entry points for each supported dataset family. These are operational concerns rather than architectural cores, but they determine how the harness is deployed and exercised.

Main entry points: [`agent_harness/core/runtime/loop/agent_loop.py`](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/agent_loop.py) (Agent loop), [`benchmarks/runner`](https://github.com/apodexai/agentharness/tree/main/benchmarks/runner) (Runner), [`workflows/react_base/spec.py`](https://github.com/apodexai/agentharness/blob/main/workflows/react_base/spec.py) (Workflow spec), [`agent_harness/core/runtime/loop/tool_exec.py`](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/tool_exec.py) (Tool exec)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmarks["Benchmarks"]
  node_bench_registry["Benchmark reg<br/>[registry.py]"]
  node_task_gen["Task gen"]
  node_answer_ext["Answer parse<br/>answer extraction"]
  node_judge_set["Judges"]
  node_runner["Runner"]
end

subgraph group_workflow["Workflow"]
  node_workflow_spec["Workflow spec<br/>[spec.py]"]
  node_main_agent["Main agent<br/>agent node<br/>[main_agent.py]"]
  node_profiles["Profiles"]
  node_wf_obs["Workflow obs"]
end

subgraph group_runtime["Runtime"]
  node_agent_loop["Agent loop<br/>runtime loop<br/>[agent_loop.py]"]
  node_tool_parse["Tool parse<br/>tool call parsing"]
  node_tool_exec["Tool exec<br/>[tool_exec.py]"]
  node_context_ctrl["Context ctrl<br/>context budget<br/>[context_budget.py]"]
  node_event_bus["Event bus<br/>runtime events<br/>[bus.py]"]
  node_resources["Resource mgr<br/>permissions manager<br/>[manager.py]"]
end

subgraph group_integration["Integrations"]
  node_llm_adapter["LLM adapter<br/>model adapter<br/>[llm_adapter.py]"]
  node_llm_clients["Provider clients<br/>llm clients"]
  node_plugins["Tool plugins<br/>external tools"]
end

subgraph group_control["Control Plane"]
  node_scheduler["Scheduler<br/>[scheduler.py]"]
  node_event_store[("Event store<br/>sqlite trace store<br/>[sqlite.py]")]
end

node_bench_registry -->|"defines tasks"| node_task_gen
node_task_gen -->|"dispatches"| node_runner
node_runner -->|"loads workflow"| node_scheduler
node_scheduler -->|"selects"| node_workflow_spec
node_workflow_spec -->|"contains"| node_main_agent
node_workflow_spec -->|"tuned by"| node_profiles
node_main_agent -->|"drives"| node_agent_loop
node_agent_loop -->|"calls model"| node_llm_adapter
node_llm_adapter -->|"uses client"| node_llm_clients
node_agent_loop -->|"parses calls"| node_tool_parse
node_tool_parse -->|"executes"| node_tool_exec
node_tool_exec -->|"invokes"| node_plugins
node_agent_loop -->|"limits context"| node_context_ctrl
node_agent_loop -->|"emits"| node_event_bus
node_event_bus -->|"observed by"| node_wf_obs
node_wf_obs -.->|"modifies flow"| node_agent_loop
node_tool_exec -->|"checked by"| node_resources
node_event_bus -->|"persists"| node_event_store
node_runner -->|"extracts answers"| node_answer_ext
node_answer_ext -->|"scores"| node_judge_set
node_judge_set -->|"aggregates"| node_runner

click node_bench_registry "https://github.com/apodexai/agentharness/blob/main/benchmarks/core/registry.py"
click node_task_gen "https://github.com/apodexai/agentharness/blob/main/benchmarks/core/harbor_task_generator.py"
click node_answer_ext "https://github.com/apodexai/agentharness/blob/main/benchmarks/core/answer_extractor.py"
click node_judge_set "https://github.com/apodexai/agentharness/tree/main/benchmarks/judges"
click node_runner "https://github.com/apodexai/agentharness/tree/main/benchmarks/runner"
click node_workflow_spec "https://github.com/apodexai/agentharness/blob/main/workflows/react_base/spec.py"
click node_main_agent "https://github.com/apodexai/agentharness/blob/main/workflows/react_base/nodes/main_agent.py"
click node_profiles "https://github.com/apodexai/agentharness/tree/main/workflows/react_base/profiles"
click node_wf_obs "https://github.com/apodexai/agentharness/tree/main/workflows/react_base/observers"
click node_agent_loop "https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/agent_loop.py"
click node_tool_parse "https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/tool_call_parser.py"
click node_tool_exec "https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/tool_exec.py"
click node_context_ctrl "https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/context_budget.py"
click node_event_bus "https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/events/bus.py"
click node_resources "https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/resources/manager.py"
click node_llm_adapter "https://github.com/apodexai/agentharness/blob/main/agent_harness/infra/llm_adapter.py"
click node_llm_clients "https://github.com/apodexai/agentharness/tree/main/agent_harness/infra"
click node_plugins "https://github.com/apodexai/agentharness/tree/main/plugins/tools"
click node_scheduler "https://github.com/apodexai/agentharness/blob/main/agent_harness/scheduling/scheduler.py"
click node_event_store "https://github.com/apodexai/agentharness/blob/main/agent_harness/state/event_store/sqlite.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bench_registry,node_task_gen,node_answer_ext,node_judge_set,node_runner toneBlue
class node_workflow_spec,node_main_agent,node_profiles,node_wf_obs toneAmber
class node_agent_loop,node_tool_parse,node_tool_exec,node_context_ctrl,node_event_bus,node_resources toneMint
class node_llm_adapter,node_llm_clients,node_plugins toneRose
class node_scheduler,node_event_store toneIndigo
```

## Components

### Benchmarks

Dataset families, task generation, judges, and benchmark runners.

- [Benchmark reg](https://github.com/apodexai/agentharness/blob/main/benchmarks/core/registry.py): benchmark registry · Registers benchmark families and resolves benchmark-specific task/judge wiring. · `benchmarks/core/registry.py`
- [Task gen](https://github.com/apodexai/agentharness/blob/main/benchmarks/core/harbor_task_generator.py): task generation · Builds benchmark tasks and questions from dataset inputs. · `benchmarks/core/harbor_task_generator.py`
- [Answer parse](https://github.com/apodexai/agentharness/blob/main/benchmarks/core/answer_extractor.py): answer extraction · Extracts benchmark answers from agent outputs for scoring. · `benchmarks/core/answer_extractor.py`
- [Judges](https://github.com/apodexai/agentharness/tree/main/benchmarks/judges): scoring judges · Benchmark-specific evaluation logic for correctness and accuracy aggregation. · `benchmarks/judges`
- [Runner](https://github.com/apodexai/agentharness/tree/main/benchmarks/runner): benchmark runner · Launches isolated per-task executions, checks progress, and rejudges runs. · `benchmarks/runner`

### Workflow

Canonical ReAct workflow configuration and its control observers.

- [Workflow spec](https://github.com/apodexai/agentharness/blob/main/workflows/react_base/spec.py): workflow spec · Defines the ReAct workflow shape and node wiring. · `workflows/react_base/spec.py`
- [Main agent](https://github.com/apodexai/agentharness/blob/main/workflows/react_base/nodes/main_agent.py): agent node · Canonical ReAct agent node used for benchmark runs. · `workflows/react_base/nodes/main_agent.py`
- [Profiles](https://github.com/apodexai/agentharness/tree/main/workflows/react_base/profiles): runtime profiles · YAML profiles that tune context, compaction, and runtime behavior. · `workflows/react_base/profiles`
- [Workflow obs](https://github.com/apodexai/agentharness/tree/main/workflows/react_base/observers): workflow observers · Normalizes tool calls and rolls back duplicate, empty, refusal, and trajectory cases. · `workflows/react_base/observers`

### Runtime

Core execution engine for agent loops, events, tools, and resource control.

- [Agent loop](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/agent_loop.py): runtime loop · Drives the iterative think-act-observe execution cycle. · `agent_harness/core/runtime/loop/agent_loop.py`
- [Tool parse](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/tool_call_parser.py): tool call parsing · Parses model tool-call output into executable actions. · `agent_harness/core/runtime/loop/tool_call_parser.py`
- [Tool exec](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/tool_exec.py): tool execution · Executes tool requests through the runtime boundary. · `agent_harness/core/runtime/loop/tool_exec.py`
- [Context ctrl](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/loop/context_budget.py): context budget · Manages context windows, compaction, and pause checks. · `agent_harness/core/runtime/loop/context_budget.py`
- [Event bus](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/events/bus.py): runtime events · Publishes runtime events for observers, logging, and trajectory capture. · `agent_harness/core/runtime/events/bus.py`
- [Resource mgr](https://github.com/apodexai/agentharness/blob/main/agent_harness/core/runtime/resources/manager.py): permissions manager · Controls tool and LLM resource permissions inside the runtime. · `agent_harness/core/runtime/resources/manager.py`

### Integrations

Model adapters and external tool plugins.

- [LLM adapter](https://github.com/apodexai/agentharness/blob/main/agent_harness/infra/llm_adapter.py): model adapter · Wraps provider clients behind an OpenAI-style chat/tool-call interface. · `agent_harness/infra/llm_adapter.py`
- [Provider clients](https://github.com/apodexai/agentharness/tree/main/agent_harness/infra): llm clients · OpenAI- and Anthropic-compatible clients used by the harness. · `agent_harness/infra`
- [Tool plugins](https://github.com/apodexai/agentharness/tree/main/plugins/tools): external tools · Search, fetch, and Python execution tools exposed to the agent. · `plugins/tools`

### Control Plane

Scheduling, process orchestration, and persisted run state.

- [Scheduler](https://github.com/apodexai/agentharness/blob/main/agent_harness/scheduling/scheduler.py): workflow scheduler · Loads workflows, registers pipelines, and coordinates runs. · `agent_harness/scheduling/scheduler.py`
- [Event store](https://github.com/apodexai/agentharness/blob/main/agent_harness/state/event_store/sqlite.py): sqlite trace store · Persists local run traces for replay and analysis. · `agent_harness/state/event_store/sqlite.py`

## Connections

- Benchmark reg → Task gen: defines tasks · Benchmark registry selects the task generation path for a family.
- Task gen → Runner: dispatches · Generated tasks are handed to the benchmark runner.
- Runner → Scheduler: loads workflow · Runner asks the scheduler to resolve and launch the configured pipeline.
- Scheduler → Workflow spec: selects · Workflow loading resolves the ReAct workflow specification.
- Workflow spec → Main agent: contains · The spec points to the main agent node as the central workflow component.
- Workflow spec → Profiles: tuned by · Profiles modify runtime behavior without changing the workflow shape.
- Main agent → Agent loop: drives · The agent node is executed by the core runtime loop.
- Agent loop → LLM adapter: calls model · The loop sends model requests through the LLM adapter boundary.
- LLM adapter → Provider clients: uses client · The adapter delegates to provider-specific model clients.
- Agent loop → Tool parse: parses calls · Tool-call outputs are parsed before execution.
- Tool parse → Tool exec: executes · Parsed tool actions are executed by the runtime.
- Tool exec → Tool plugins: invokes · External integrations are performed through plugin tools.
- Agent loop → Context ctrl: limits context · Context budgeting and compaction govern loop continuation.
- Agent loop → Event bus: emits · Runtime steps produce events for observers and persistence.
- Event bus → Workflow obs: observed by · Workflow observers consume runtime events as side channels.
- Workflow obs → Agent loop: modifies flow · Observers can normalize, roll back, or annotate ongoing execution.
- Tool exec → Resource mgr: checked by · Tool access is mediated by the resource and permission manager.
- Event bus → Event store: persists · Run traces are recorded into the local SQLite event store.
- Runner → Answer parse: extracts answers · Benchmark outputs are normalized into judge-ready answers.
- Answer parse → Judges: scores · Extracted answers are evaluated by benchmark-specific judges.
- Judges → Runner: aggregates · Judges return task results for accuracy aggregation and reporting.