How stanford-iris-lab/meta-harness-tbench2-artifact works
1. This repository is a thin Python agent harness, not a full application. Its job is to plug a custom agent into the Harbor/Terminus benchmark runtime for Terminal-Bench 2.0.
2. The core entry point is agent.py, which exposes the agent class used by Harbor (AgentHarness per the README). This is the main integration boundary between the repo and the external evaluation framework.
3. The runtime control flow is: Harbor launches the benchmark environment, imports agent:AgentHarness, and then drives the agent loop against Terminal-Bench tasks. The agent receives task context, interacts with the sandbox, and returns actions through the framework’s runloop.
4. The architecture is centered on a single LLM-backed decision loop. Model calls go to Anthropic Claude via the Harbor runner, with the API key supplied externally through ANTHROPIC_API_KEY.
5. A key repo-specific behavior is environment bootstrapping. Before the normal agent loop begins, the harness gathers a snapshot of the sandbox state—current working directory, file listing, available languages/tools, package managers, and memory—and injects that into the initial prompt.
6. That bootstrap step is the main data-flow enhancement in this repo: sandbox metadata flows into prompt construction, then into the model, reducing the need for the agent to spend early turns discovering basic environment facts.
7. Prompting is separated into template assets under prompt-templates/. The terminus-kira.txt template likely contains the system/instruction prompt inherited from Terminus-KIRA, with the harness adding environment-specific context on top.
8. anthropic_caching.py is a supporting subsystem for API efficiency and/or prompt reuse. It likely wraps Anthropic requests or message formatting to support cached prompt prefixes, which matters for repeated benchmark runs.
9. The project boundary is intentionally narrow: it does not implement the benchmark, terminal sandbox, or model orchestration itself. Those responsibilities live in Harbor and Terminal-Bench; this repo only customizes agent behavior and prompt initialization.
10. The external dependencies that shape the architecture are Harbor’s Terminus-2 framework, the Terminal-Bench 2.0 environment, and Anthropic’s Claude API. The repository is therefore best diagrammed as a small adapter layer between benchmark runner, sandbox environment, and LLM provider.
11. pyproject.toml defines the Python package/runtime metadata and dependencies, making this a standard Python-based harness distributed as code rather than as a standalone service.
12. In diagram terms, the important components are: Harbor benchmark runner, Terminal-Bench sandbox, this agent harness, prompt templates/cache helper, and Anthropic Claude. The main arrows are sandbox state into prompt construction, prompt into model calls, and model outputs back into terminal actions.
Main entry points: agent.py Decision loop · anthropic_caching.py Caching helper · terminus-kira.txt Terminus prompt · pyproject.toml pyproject.toml
Components 11
This repo
Thin Python harness that adapts Harbor/Terminus to Terminal-Bench 2.0
- agent.py · Repo boundary exposed to Harbor as AgentHarness
agent.py - AgentHarness · Main integration class that connects Harbor tasks to the repo's decision loop
agent.py - Bootstrap · Collects cwd, files, tools, package managers, languages, and memory before the first turn
agent.py - Prompt builder · Combines template instructions with live sandbox context
agent.py - Decision loop · Turns task context and prompts into terminal actions across benchmark turns
agent.py - Caching helper · Supports Anthropic prompt reuse or cached message formatting
anthropic_caching.py - Terminus prompt · Base system/instruction prompt inherited from Terminus-KIRA
prompt-templates/terminus-kira.txt - pyproject.toml · Defines Python package metadata and runtime dependencies
pyproject.toml
External systems
Benchmark runtime, sandbox, and model provider
- Harbor runner · Launches the benchmark, imports the agent entrypoint, and drives the runloop
- Terminal-Bench · Provides the terminal tasks, sandbox state, and action execution surface
- Claude API · Anthropic model endpoint used for decision making
Connections 12
- Harbor runner → agent.py: imports · Harbor loads the agent entrypoint from this module
- agent.py → AgentHarness: exposes · Entry module publishes the AgentHarness class
- AgentHarness → Bootstrap: snapshots · Harness gathers sandbox metadata before starting the loop
- Bootstrap → Prompt builder: injects · Environment facts are added to the initial prompt
- Terminus prompt → Prompt builder: base prompt · Template provides the inherited instruction scaffold
- Prompt builder → Decision loop: feeds · Assembled prompt becomes input to the agent loop
- Decision loop → Claude API: calls · The loop sends prompts to Anthropic Claude through Harbor
- Claude API → Decision loop: returns output · Model responses are turned into actions or next-step reasoning
- Decision loop → Terminal-Bench: acts in sandbox · Generated actions are executed against the terminal environment
- Terminal-Bench → Bootstrap: provides state · Sandbox state is the source of the bootstrap snapshot
- Caching helper → Decision loop: optimizes · Helper supports cached prompt/message handling in repeated runs
- AgentHarness → pyproject.toml: packaged by · Project metadata defines how the harness is installed and run