How harbor-framework/terminal-bench works
1. What this repo is
- terminal-bench is a benchmark platform for evaluating AI agents in real terminal environments.
- The architecture splits into two major planes: a task corpus (benchmark content) and an execution platform (runner, agent adapters, sandbox, evaluation, and reporting).
2. Core package boundary: terminal_bench/
- This is the runtime library used by the CLI and harness.
- It contains the main subsystems for agents, harness execution, terminal control, parsers, dataset/registry access, LLM integration, database/helpers, and utility code.
- The package is the central boundary that ties all other parts together.
3. Benchmark task model
- Tasks are stored as self-contained directories under original-tasks/ and adapter-generated templates under adapters/*/template.
- A task typically includes:
- a human-readable instruction,
- a task.yaml metadata file,
- a container setup (Dockerfile, docker-compose.yaml),
- a run-tests.sh script,
- and oracle/reference solution files.
- Test output files and hidden assets are often included, making the task an isolated, reproducible unit.
4. Task families and adapters
- adapters/ contains per-benchmark adapter packages such as swebench, mlebench, appworld, cybench, deveval, swelancer, sweperf, USACO, and others.
- Each adapter translates an external benchmark format into Terminal-Bench’s task schema and templates.
- This is an important ingestion boundary: external benchmark sources are normalized into a common internal task representation.
5. Execution harness
- terminal_bench/harness/ implements the orchestration of a run: launch task environment, attach an agent, collect outputs, and determine completion.
- The harness is the main runtime controller and sits between the agent layer and the terminal sandbox.
- Test execution and scoring are separate from the agent itself, which keeps evaluation deterministic and isolated from model behavior.
6. Terminal sandbox and container orchestration
- terminal_bench/terminal/ manages the actual interactive environment, including Docker Compose orchestration and tmux session handling.
- This is where the benchmark task is physically executed: the agent interacts with a terminal session inside a containerized sandbox.
- The repo also has dedicated Docker assets under docker/, including:
- base images,
- agent images,
- and an MCP server image.
- This indicates that reproducible containerization is a first-class architectural constraint.
7. Agent abstraction layer
- terminal_bench/agents/ defines a common agent interface plus several implementations:
- simple/local agents,
- oracle and null agents,
- installed CLI agents such as Claude Code, Aider, Codex, Cursor, Gemini CLI, Goose, Grok CLI, OpenHands, Qwen Code, Opencode, and Mini SWE.
- The agent layer is intentionally pluggable: the harness interacts with a standardized agent contract, while each agent adapter handles its own startup, prompt formatting, and tool wiring.
8. LLM integration
- terminal_bench/llms/ abstracts model access and chat-style interaction.
- This layer is separate from the terminal runtime so the platform can swap between providers or caching modes without changing task execution logic.
- In architecture terms, it is the model-provider boundary that feeds prompts into agents or agent wrappers.
9. Evaluation and parsing
- terminal_bench/parsers/ contains benchmark-specific result parsers such as pytest, SWE-bench, SWE-lancer, SWE-perf, and MLE-bench parsers.
- These parsers convert raw task outputs into structured pass/fail or metric results.
- The evaluation path is therefore: task execution → log/output collection → parser interpretation → result aggregation.
10. Dataset, registry, and metadata
- terminal_bench/dataset/ and terminal_bench/registry/ handle dataset packaging, metadata, and registry access.
- registry.json is the repo-level source of task/dataset indexing.
- This layer supports versioned benchmark selection such as the “terminal-bench-core” dataset mentioned in the README.
11. Reporting, dashboard, and persistence
- dashboard/ contains a separate app for browsing benchmark data, backed by data_access.py, db_init.py, and migrations.py.
- This is the repo’s persistence/reporting boundary: execution results can be stored and visualized outside the harness itself.
- The presence of database migration tooling indicates the dashboard has its own schema lifecycle.
12. CLI and developer workflow
- terminal_bench/cli/ provides the tb command entry points and operational commands for running, resuming, inspecting, and administering runs.
- scripts_bash/ and scripts_python/ contain repo-wide automation for validation, task checks, harness execution, and dataset export.
- This layer is the operational interface for contributors and benchmark operators.
13. CI and validation gates
- .github/workflows/ enforces task integrity, Dockerfile sanity, test references, task field validation, and benchmark-specific checks.
- These checks matter architecturally because they preserve task reproducibility and prevent broken benchmark artifacts from entering the dataset.
14. External dependencies and integration points
- Material external dependencies include Docker/Docker Compose, tmux, LLM providers, and some benchmark-specific services such as Supabase uploads or dashboard storage.
- The system is designed so those dependencies are isolated behind adapters, harness code, or workflow scripts rather than mixed into task logic.
15. Overall data flow
- A benchmark task enters through an adapter or lives in original-tasks/.
- The CLI selects a dataset and launches the harness.
- The harness provisions a containerized terminal, instantiates an agent, and streams interaction through the terminal session.
- After the agent finishes, parsers and tests evaluate the outputs, and results can be stored for dashboarding, registry use, or leaderboard submission.
Components 18
Task corpus
Benchmark content, templates, and normalization adapters.
- Original tasks · Self-contained benchmark tasks with instructions, containers, and tests.
original-tasks - Adapters · Per-benchmark importers that normalize external task formats into Terminal-Bench templates.
adapters
Execution platform
CLI, harness, agents, terminal sandbox, parsing, and model integration.
- Core package · Main library boundary shared by CLI, harness, agents, terminal, parsers, and LLM code.
terminal_bench - TB CLI · User-facing commands for runs, datasets, admin, cache, and task operations.
terminal_bench/cli/tb - Harness · Coordinates agent execution, environment lifecycle, and run completion.
terminal_bench/harness - Terminal · Docker Compose and tmux-backed interactive terminal environment.
terminal_bench/terminal - Agents · Common agent interface plus local, oracle, null, MCP, and installed CLI agents.
terminal_bench/agents - Installed CLIs · Wrappers for external CLI agents such as Claude Code, Aider, Codex, Cursor, Gemini, Goose, Grok, OpenHands, Qwen Code, and others.
terminal_bench/agents/installed_agents - LLMs · Chat-style model provider abstraction and provider-specific wiring.
terminal_bench/llms - Parsers · Benchmark-specific output interpreters that produce structured scores and pass/fail results.
terminal_bench/parsers - Docker assets · Base images, agent images, and MCP server assets used to build reproducible environments.
docker
Ops & reporting
Dataset/registry, dashboard persistence, scripts, and validation gates.
- Dataset · Dataset packaging and metadata model used for benchmark selection and export.
terminal_bench/dataset - Registry · Task and dataset registry access backed by repo-level indexing data.
terminal_bench/registry - Dashboard · Persistence and browsing app for benchmark results and stored runs.
dashboard - Bash scripts · Operational scripts for validation, harness execution, image handling, and task checks.
scripts_bash - Python scripts · Repo automation for harness execution, task export, and output aggregation.
scripts_python - CI workflows · GitHub Actions checks that enforce task integrity, Docker sanity, and benchmark consistency.
.github/workflows - Registry file · Repo-level source of task and dataset indexing.
registry.json
Connections 24
- Adapters → Original tasks: generate tasks · Adapters translate external benchmarks into benchmark task directories.
- TB CLI → Dataset: selects · CLI chooses datasets and run targets from metadata.
- TB CLI → Harness: launches · CLI commands start and manage harness runs.
- Harness → Agents: instantiates · Harness creates or attaches the selected agent implementation.
- Harness → Terminal: controls · Harness drives the interactive containerized terminal environment.
- Harness → Parsers: evaluates · Harness hands run output to parsers for structured scoring.
- Agents → LLMs: uses · Agent implementations call model providers or chat abstractions when needed.
- Installed CLIs → Docker assets: built from · Installed CLI agents rely on containerized images and setup scripts.
- Terminal → Docker assets: orchestrates · Terminal runtime provisions Docker Compose environments and supporting images.
- Parsers → Registry file: indexes · Parsed results feed registry-backed benchmark organization and lookup.
- Dataset → Registry: resolved by · Dataset selection is backed by registry access.
- Dashboard → Registry file: reads · Dashboard/reporting uses indexed benchmark metadata and stored results.
- Bash scripts → CI workflows: supports · Repo scripts mirror or back workflow validations.
- Python scripts → Harness: runs · Python automation wraps harness execution and output aggregation.
- CI workflows → Original tasks: validates · CI checks task structure, references, and reproducibility constraints.
- CI workflows → Adapters: checks · Workflows validate adapter-generated task artifacts and invariants.
- Core package → TB CLI: contains · CLI lives within the core runtime package boundary.
- Core package → Harness: contains · Harness implementation is part of the core package.
- Core package → Terminal: contains · Terminal control is implemented inside the core package.
- Core package → Agents: contains · Agent abstractions and implementations live in the core package.
- Core package → LLMs: contains · LLM integration is a core package subsystem.
- Core package → Parsers: contains · Parser implementations are part of the core package.
- Core package → Dataset: contains · Dataset helpers and metadata live in the core package.
- Core package → Registry: contains · Registry client is part of the core package.