How apodexai/agentharness works
1. What this repo is
AgentHarness is a Python 3.12 evaluation harness for running deep-research agents against public benchmarks. It is not a product app; it is an execution framework that loads a benchmark task, runs an agent workflow, captures events, and scores the output with benchmark-specific judges.
2. Top-level execution flow
The main path is: benchmark dataset task → benchmark runner → subprocess-per-question execution → selected workflow/pipeline → agent loop with model calls and tools → event/log capture → judge/accuracy aggregation. The subprocess boundary is important: each question is isolated for reproducibility, hang recovery, and independent reruns.
3. Benchmarking subsystem
The benchmarks/ package is the orchestration layer for experiments. It contains dataset/family definitions, task generation, answer extraction, judge logic, and runners such as run_subprocess.py and check_progress.py. This is where benchmark-specific inputs and scoring live, separate from the agent runtime itself.
4. Workflow layer
The workflows/react_base/ package defines the canonical ReAct-style agent configuration used in the paper. It includes the main agent node, prompt/spec/profile definitions, and workflow-specific observers and compactors. Profiles like default.yaml and keep5.yaml alter runtime behavior without changing the core engine.
5. Core runtime engine
The agent_harness/core/runtime/ tree is the central execution engine. It contains the agent loop, tool-call parsing, tool execution, context-budget handling, compaction, pause checks, event bus, DAG utilities, registries, and resource/permission management. This is the boundary between “workflow description” and “actual runtime behavior.”
6. Model and LLM integration
LLM access is abstracted through agent_harness/core/llm.py, infra/llm_adapter.py, and provider clients for OpenAI- and Anthropic-compatible endpoints. In practice, the README’s default path points OPENAI_BASE_URL/OPENAI_API_KEY/OPENAI_MODEL at a model server such as SGLang. The runtime is designed around OpenAI-style chat/tool-call interfaces, with model-profile logic controlling parsing and context behavior.
7. Tooling and external services
Tools are first-class, but are isolated in plugins/tools/ and permissioned through runtime resource managers. The concrete external integrations are web search, web fetch, and Python code execution, backed by services like Serper, Jina, and E2B. This creates a clear boundary: the agent can request tools, but execution is mediated by the runtime and permissions layer.
8. Observers and control signals
Observers appear in both agent_harness/components/observers/ and workflows/react_base/observers/. They are side-channel processors that watch runtime events and enforce or react to conditions such as budget limits, context-size guards, rollback on duplicate/empty/refusal cases, tool-call normalization, trajectory capture, and SSE/console output. They do not own the agent loop; they modify or annotate it.
9. State and event persistence
The state layer is intentionally lightweight: agent_harness/state/event_store/sqlite.py provides a local event store for recording run traces. This supports debugging, replay, and post-run analysis without coupling the runtime to a larger database architecture.
10. Scheduling and process management
The agent_harness/scheduling/ package manages workflow loading, pipeline registration, and process orchestration. Combined with the benchmark runner, this is the experiment-control plane: it resolves which pipeline/workflow to run, launches work, and coordinates multi-run evaluation.
11. Data model and shared types
The models/ and core/ packages define the shared contracts: agent definitions, tasks, pipeline specs, node context, events, messages, errors, and protocol interfaces. These types are the internal API between benchmarks, workflows, runtime, and observers, and are key to understanding the repo boundary structure.
12. Operational dependencies
The repo assumes several external runtime inputs: benchmark datasets on disk, an OpenAI-compatible model endpoint, and optional web/code service credentials from .env. CI exists via GitHub Actions, and benchmark scripts in scripts/ provide canned entry points for each supported dataset family. These are operational concerns rather than architectural cores, but they determine how the harness is deployed and exercised.
Main entry points: agent_loop.py Agent loop · runner Runner · spec.py Workflow spec · tool_exec.py Tool exec
Components 20
Benchmarks
Dataset families, task generation, judges, and benchmark runners.
- Benchmark reg · Registers benchmark families and resolves benchmark-specific task/judge wiring.
benchmarks/core/registry.py - Task gen · Builds benchmark tasks and questions from dataset inputs.
benchmarks/core/harbor_task_generator.py - Answer parse · Extracts benchmark answers from agent outputs for scoring.
benchmarks/core/answer_extractor.py - Judges · Benchmark-specific evaluation logic for correctness and accuracy aggregation.
benchmarks/judges - Runner · Launches isolated per-task executions, checks progress, and rejudges runs.
benchmarks/runner
Workflow
Canonical ReAct workflow configuration and its control observers.
- Workflow spec · Defines the ReAct workflow shape and node wiring.
workflows/react_base/spec.py - Main agent · Canonical ReAct agent node used for benchmark runs.
workflows/react_base/nodes/main_agent.py - Profiles · YAML profiles that tune context, compaction, and runtime behavior.
workflows/react_base/profiles - Workflow obs · Normalizes tool calls and rolls back duplicate, empty, refusal, and trajectory cases.
workflows/react_base/observers
Runtime
Core execution engine for agent loops, events, tools, and resource control.
- Agent loop · Drives the iterative think-act-observe execution cycle.
agent_harness/core/runtime/loop/agent_loop.py - Tool parse · Parses model tool-call output into executable actions.
agent_harness/core/runtime/loop/tool_call_parser.py - Tool exec · Executes tool requests through the runtime boundary.
agent_harness/core/runtime/loop/tool_exec.py - Context ctrl · Manages context windows, compaction, and pause checks.
agent_harness/core/runtime/loop/context_budget.py - Event bus · Publishes runtime events for observers, logging, and trajectory capture.
agent_harness/core/runtime/events/bus.py - Resource mgr · Controls tool and LLM resource permissions inside the runtime.
agent_harness/core/runtime/resources/manager.py
Integrations
Model adapters and external tool plugins.
- LLM adapter · Wraps provider clients behind an OpenAI-style chat/tool-call interface.
agent_harness/infra/llm_adapter.py - Provider clients · OpenAI- and Anthropic-compatible clients used by the harness.
agent_harness/infra - Tool plugins · Search, fetch, and Python execution tools exposed to the agent.
plugins/tools
Control Plane
Scheduling, process orchestration, and persisted run state.
- Scheduler · Loads workflows, registers pipelines, and coordinates runs.
agent_harness/scheduling/scheduler.py - Event store · Persists local run traces for replay and analysis.
agent_harness/state/event_store/sqlite.py
Connections 21
- Benchmark reg → Task gen: defines tasks · Benchmark registry selects the task generation path for a family.
- Task gen → Runner: dispatches · Generated tasks are handed to the benchmark runner.
- Runner → Scheduler: loads workflow · Runner asks the scheduler to resolve and launch the configured pipeline.
- Scheduler → Workflow spec: selects · Workflow loading resolves the ReAct workflow specification.
- Workflow spec → Main agent: contains · The spec points to the main agent node as the central workflow component.
- Workflow spec → Profiles: tuned by · Profiles modify runtime behavior without changing the workflow shape.
- Main agent → Agent loop: drives · The agent node is executed by the core runtime loop.
- Agent loop → LLM adapter: calls model · The loop sends model requests through the LLM adapter boundary.
- LLM adapter → Provider clients: uses client · The adapter delegates to provider-specific model clients.
- Agent loop → Tool parse: parses calls · Tool-call outputs are parsed before execution.
- Tool parse → Tool exec: executes · Parsed tool actions are executed by the runtime.
- Tool exec → Tool plugins: invokes · External integrations are performed through plugin tools.
- Agent loop → Context ctrl: limits context · Context budgeting and compaction govern loop continuation.
- Agent loop → Event bus: emits · Runtime steps produce events for observers and persistence.
- Event bus → Workflow obs: observed by · Workflow observers consume runtime events as side channels.
- Workflow obs → Agent loop: modifies flow · Observers can normalize, roll back, or annotate ongoing execution.
- Tool exec → Resource mgr: checked by · Tool access is mediated by the resource and permission manager.
- Event bus → Event store: persists · Run traces are recorded into the local SQLite event store.
- Runner → Answer parse: extracts answers · Benchmark outputs are normalized into judge-ready answers.
- Answer parse → Judges: scores · Extracted answers are evaluated by benchmark-specific judges.
- Judges → Runner: aggregates · Judges return task results for accuracy aggregation and reporting.