harbor-framework/terminal-bench

1. What this repo is

- terminal-bench is a benchmark platform for evaluating AI agents in real terminal environments.

- The architecture splits into two major planes: a task corpus (benchmark content) and an execution platform (runner, agent adapters, sandbox, evaluation, and reporting).

2. Core package boundary: terminal_bench/

- This is the runtime library used by the CLI and harness.

- It contains the main subsystems for agents, harness execution, terminal control, parsers, dataset/registry access, LLM integration, database/helpers, and utility code.

- The package is the central boundary that ties all other parts together.

3. Benchmark task model

- Tasks are stored as self-contained directories under original-tasks/ and adapter-generated templates under adapters/*/template.

- A task typically includes:

- a human-readable instruction,

- a task.yaml metadata file,

- a container setup (Dockerfile, docker-compose.yaml),

- a run-tests.sh script,

- and oracle/reference solution files.

- Test output files and hidden assets are often included, making the task an isolated, reproducible unit.

4. Task families and adapters

- adapters/ contains per-benchmark adapter packages such as swebench, mlebench, appworld, cybench, deveval, swelancer, sweperf, USACO, and others.

- Each adapter translates an external benchmark format into Terminal-Bench’s task schema and templates.

- This is an important ingestion boundary: external benchmark sources are normalized into a common internal task representation.

5. Execution harness

- terminal_bench/harness/ implements the orchestration of a run: launch task environment, attach an agent, collect outputs, and determine completion.

- The harness is the main runtime controller and sits between the agent layer and the terminal sandbox.

- Test execution and scoring are separate from the agent itself, which keeps evaluation deterministic and isolated from model behavior.

6. Terminal sandbox and container orchestration

- terminal_bench/terminal/ manages the actual interactive environment, including Docker Compose orchestration and tmux session handling.

- This is where the benchmark task is physically executed: the agent interacts with a terminal session inside a containerized sandbox.

- The repo also has dedicated Docker assets under docker/, including:

- base images,

- agent images,

- and an MCP server image.

- This indicates that reproducible containerization is a first-class architectural constraint.

7. Agent abstraction layer

- terminal_bench/agents/ defines a common agent interface plus several implementations:

- simple/local agents,

- oracle and null agents,

- installed CLI agents such as Claude Code, Aider, Codex, Cursor, Gemini CLI, Goose, Grok CLI, OpenHands, Qwen Code, Opencode, and Mini SWE.

- The agent layer is intentionally pluggable: the harness interacts with a standardized agent contract, while each agent adapter handles its own startup, prompt formatting, and tool wiring.

8. LLM integration

- terminal_bench/llms/ abstracts model access and chat-style interaction.

- This layer is separate from the terminal runtime so the platform can swap between providers or caching modes without changing task execution logic.

- In architecture terms, it is the model-provider boundary that feeds prompts into agents or agent wrappers.

9. Evaluation and parsing

- terminal_bench/parsers/ contains benchmark-specific result parsers such as pytest, SWE-bench, SWE-lancer, SWE-perf, and MLE-bench parsers.

- These parsers convert raw task outputs into structured pass/fail or metric results.

- The evaluation path is therefore: task execution → log/output collection → parser interpretation → result aggregation.

10. Dataset, registry, and metadata

- terminal_bench/dataset/ and terminal_bench/registry/ handle dataset packaging, metadata, and registry access.

- registry.json is the repo-level source of task/dataset indexing.

- This layer supports versioned benchmark selection such as the “terminal-bench-core” dataset mentioned in the README.

11. Reporting, dashboard, and persistence

- dashboard/ contains a separate app for browsing benchmark data, backed by data_access.py, db_init.py, and migrations.py.

- This is the repo’s persistence/reporting boundary: execution results can be stored and visualized outside the harness itself.

- The presence of database migration tooling indicates the dashboard has its own schema lifecycle.

12. CLI and developer workflow

- terminal_bench/cli/ provides the tb command entry points and operational commands for running, resuming, inspecting, and administering runs.

- scripts_bash/ and scripts_python/ contain repo-wide automation for validation, task checks, harness execution, and dataset export.

- This layer is the operational interface for contributors and benchmark operators.

13. CI and validation gates

- .github/workflows/ enforces task integrity, Dockerfile sanity, test references, task field validation, and benchmark-specific checks.

- These checks matter architecturally because they preserve task reproducibility and prevent broken benchmark artifacts from entering the dataset.

14. External dependencies and integration points

- Material external dependencies include Docker/Docker Compose, tmux, LLM providers, and some benchmark-specific services such as Supabase uploads or dashboard storage.

- The system is designed so those dependencies are isolated behind adapters, harness code, or workflow scripts rather than mixed into task logic.

15. Overall data flow

- A benchmark task enters through an adapter or lives in original-tasks/.

- The CLI selects a dataset and launches the harness.

- The harness provisions a containerized terminal, instantiates an agent, and streams interaction through the terminal session.

- After the agent finishes, parsers and tests evaluate the outputs, and results can be stored for dashboarding, registry use, or leaderboard submission.