# harbor-framework/terminal-bench architecture
> Architecture diagram of the GitHub repository harbor-framework/terminal-bench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.
- Interactive diagram: https://gitdiagram.com/harbor-framework/terminal-bench
- Repository: https://github.com/harbor-framework/terminal-bench
- Video tour (about a minute): https://gitdiagram.com/harbor-framework/terminal-bench/video
## Overview
1. **What this repo is**
- `terminal-bench` is a benchmark platform for evaluating AI agents in real terminal environments.
- The architecture splits into two major planes: a **task corpus** (benchmark content) and an **execution platform** (runner, agent adapters, sandbox, evaluation, and reporting).
2. **Core package boundary: `terminal_bench/`**
- This is the runtime library used by the CLI and harness.
- It contains the main subsystems for **agents**, **harness execution**, **terminal control**, **parsers**, **dataset/registry access**, **LLM integration**, **database/helpers**, and **utility code**.
- The package is the central boundary that ties all other parts together.
3. **Benchmark task model**
- Tasks are stored as self-contained directories under `original-tasks/` and adapter-generated templates under `adapters/*/template`.
- A task typically includes:
- a human-readable instruction,
- a `task.yaml` metadata file,
- a container setup (`Dockerfile`, `docker-compose.yaml`),
- a `run-tests.sh` script,
- and oracle/reference solution files.
- Test output files and hidden assets are often included, making the task an isolated, reproducible unit.
4. **Task families and adapters**
- `adapters/` contains per-benchmark adapter packages such as `swebench`, `mlebench`, `appworld`, `cybench`, `deveval`, `swelancer`, `sweperf`, `USACO`, and others.
- Each adapter translates an external benchmark format into Terminal-Bench’s task schema and templates.
- This is an important ingestion boundary: external benchmark sources are normalized into a common internal task representation.
5. **Execution harness**
- `terminal_bench/harness/` implements the orchestration of a run: launch task environment, attach an agent, collect outputs, and determine completion.
- The harness is the main runtime controller and sits between the agent layer and the terminal sandbox.
- Test execution and scoring are separate from the agent itself, which keeps evaluation deterministic and isolated from model behavior.
6. **Terminal sandbox and container orchestration**
- `terminal_bench/terminal/` manages the actual interactive environment, including Docker Compose orchestration and tmux session handling.
- This is where the benchmark task is physically executed: the agent interacts with a terminal session inside a containerized sandbox.
- The repo also has dedicated Docker assets under `docker/`, including:
- base images,
- agent images,
- and an MCP server image.
- This indicates that reproducible containerization is a first-class architectural constraint.
7. **Agent abstraction layer**
- `terminal_bench/agents/` defines a common agent interface plus several implementations:
- simple/local agents,
- oracle and null agents,
- installed CLI agents such as Claude Code, Aider, Codex, Cursor, Gemini CLI, Goose, Grok CLI, OpenHands, Qwen Code, Opencode, and Mini SWE.
- The agent layer is intentionally pluggable: the harness interacts with a standardized agent contract, while each agent adapter handles its own startup, prompt formatting, and tool wiring.
8. **LLM integration**
- `terminal_bench/llms/` abstracts model access and chat-style interaction.
- This layer is separate from the terminal runtime so the platform can swap between providers or caching modes without changing task execution logic.
- In architecture terms, it is the model-provider boundary that feeds prompts into agents or agent wrappers.
9. **Evaluation and parsing**
- `terminal_bench/parsers/` contains benchmark-specific result parsers such as pytest, SWE-bench, SWE-lancer, SWE-perf, and MLE-bench parsers.
- These parsers convert raw task outputs into structured pass/fail or metric results.
- The evaluation path is therefore: task execution → log/output collection → parser interpretation → result aggregation.
10. **Dataset, registry, and metadata**
- `terminal_bench/dataset/` and `terminal_bench/registry/` handle dataset packaging, metadata, and registry access.
- `registry.json` is the repo-level source of task/dataset indexing.
- This layer supports versioned benchmark selection such as the “terminal-bench-core” dataset mentioned in the README.
11. **Reporting, dashboard, and persistence**
- `dashboard/` contains a separate app for browsing benchmark data, backed by `data_access.py`, `db_init.py`, and `migrations.py`.
- This is the repo’s persistence/reporting boundary: execution results can be stored and visualized outside the harness itself.
- The presence of database migration tooling indicates the dashboard has its own schema lifecycle.
12. **CLI and developer workflow**
- `terminal_bench/cli/` provides the `tb` command entry points and operational commands for running, resuming, inspecting, and administering runs.
- `scripts_bash/` and `scripts_python/` contain repo-wide automation for validation, task checks, harness execution, and dataset export.
- This layer is the operational interface for contributors and benchmark operators.
13. **CI and validation gates**
- `.github/workflows/` enforces task integrity, Dockerfile sanity, test references, task field validation, and benchmark-specific checks.
- These checks matter architecturally because they preserve task reproducibility and prevent broken benchmark artifacts from entering the dataset.
14. **External dependencies and integration points**
- Material external dependencies include Docker/Docker Compose, tmux, LLM providers, and some benchmark-specific services such as Supabase uploads or dashboard storage.
- The system is designed so those dependencies are isolated behind adapters, harness code, or workflow scripts rather than mixed into task logic.
15. **Overall data flow**
- A benchmark task enters through an adapter or lives in `original-tasks/`.
- The CLI selects a dataset and launches the harness.
- The harness provisions a containerized terminal, instantiates an agent, and streams interaction through the terminal session.
- After the agent finishes, parsers and tests evaluate the outputs, and results can be stored for dashboarding, registry use, or leaderboard submission.
Main entry points: [`terminal_bench`](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench) (Core package), [`terminal_bench/harness`](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/harness) (Harness), [`.github/workflows`](https://github.com/harbor-framework/terminal-bench/blob/main/.github/workflows) (CI workflows), [`terminal_bench/cli/tb`](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/cli/tb) (TB CLI)
## Diagram (Mermaid)
```mermaid
flowchart TD
subgraph group_corpus["Task corpus"]
node_original_tasks["Original tasks
task corpus"]
node_adapters["Adapters"]
end
subgraph group_runtime["Execution platform"]
node_terminal_bench_core["Core package
runtime library"]
node_cli_tb["TB CLI
command layer"]
node_harness["Harness
orchestration"]
node_terminal["Terminal
sandbox control"]
node_agents["Agents
agent layer"]
node_installed_agents["Installed CLIs
agent adapters"]
node_llms["LLMs
model access"]
node_parsers["Parsers
result parsing"]
node_docker_assets["Docker assets
container images"]
end
subgraph group_ops["Ops & reporting"]
node_dataset["Dataset"]
node_registry["Registry
indexing"]
node_dashboard["Dashboard
reporting app"]
node_scripts_bash["Bash scripts
automation"]
node_scripts_python["Python scripts
automation"]
node_github_workflows["CI workflows
validation gates
[workflows]"]
node_registry_json["Registry file
index data
[registry.json]"]
end
node_adapters -->|"generate tasks"| node_original_tasks
node_cli_tb -->|"selects"| node_dataset
node_cli_tb -->|"launches"| node_harness
node_harness -->|"instantiates"| node_agents
node_harness -->|"controls"| node_terminal
node_harness -->|"evaluates"| node_parsers
node_agents -->|"uses"| node_llms
node_installed_agents -->|"built from"| node_docker_assets
node_terminal -->|"orchestrates"| node_docker_assets
node_parsers -->|"indexes"| node_registry_json
node_dataset -->|"resolved by"| node_registry
node_dashboard -->|"reads"| node_registry_json
node_scripts_bash -->|"supports"| node_github_workflows
node_scripts_python -->|"runs"| node_harness
node_github_workflows -->|"validates"| node_original_tasks
node_github_workflows -->|"checks"| node_adapters
node_terminal_bench_core -->|"contains"| node_cli_tb
node_terminal_bench_core -->|"contains"| node_harness
node_terminal_bench_core -->|"contains"| node_terminal
node_terminal_bench_core -->|"contains"| node_agents
node_terminal_bench_core -->|"contains"| node_llms
node_terminal_bench_core -->|"contains"| node_parsers
node_terminal_bench_core -->|"contains"| node_dataset
node_terminal_bench_core -->|"contains"| node_registry
click node_original_tasks "https://github.com/harbor-framework/terminal-bench/tree/main/original-tasks"
click node_adapters "https://github.com/harbor-framework/terminal-bench/tree/main/adapters"
click node_terminal_bench_core "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench"
click node_cli_tb "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/cli/tb"
click node_harness "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/harness"
click node_terminal "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/terminal"
click node_agents "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/agents"
click node_installed_agents "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/agents/installed_agents"
click node_llms "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/llms"
click node_parsers "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/parsers"
click node_dataset "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/dataset"
click node_registry "https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/registry"
click node_dashboard "https://github.com/harbor-framework/terminal-bench/tree/main/dashboard"
click node_scripts_bash "https://github.com/harbor-framework/terminal-bench/tree/main/scripts_bash"
click node_scripts_python "https://github.com/harbor-framework/terminal-bench/tree/main/scripts_python"
click node_github_workflows "https://github.com/harbor-framework/terminal-bench/blob/main/.github/workflows"
click node_docker_assets "https://github.com/harbor-framework/terminal-bench/tree/main/docker"
click node_registry_json "https://github.com/harbor-framework/terminal-bench/blob/main/registry.json"
classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_original_tasks,node_adapters toneBlue
class node_terminal_bench_core,node_cli_tb,node_harness,node_terminal,node_agents,node_installed_agents,node_llms,node_parsers,node_docker_assets toneAmber
class node_dataset,node_registry,node_dashboard,node_scripts_bash,node_scripts_python,node_github_workflows,node_registry_json toneMint
```
## Components
### Task corpus
Benchmark content, templates, and normalization adapters.
- [Original tasks](https://github.com/harbor-framework/terminal-bench/tree/main/original-tasks): task corpus · Self-contained benchmark tasks with instructions, containers, and tests. · `original-tasks`
- [Adapters](https://github.com/harbor-framework/terminal-bench/tree/main/adapters): benchmark adapters · Per-benchmark importers that normalize external task formats into Terminal-Bench templates. · `adapters`
### Execution platform
CLI, harness, agents, terminal sandbox, parsing, and model integration.
- [Core package](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench): runtime library · Main library boundary shared by CLI, harness, agents, terminal, parsers, and LLM code. · `terminal_bench`
- [TB CLI](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/cli/tb): command layer · User-facing commands for runs, datasets, admin, cache, and task operations. · `terminal_bench/cli/tb`
- [Harness](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/harness): orchestration · Coordinates agent execution, environment lifecycle, and run completion. · `terminal_bench/harness`
- [Terminal](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/terminal): sandbox control · Docker Compose and tmux-backed interactive terminal environment. · `terminal_bench/terminal`
- [Agents](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/agents): agent layer · Common agent interface plus local, oracle, null, MCP, and installed CLI agents. · `terminal_bench/agents`
- [Installed CLIs](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/agents/installed_agents): agent adapters · Wrappers for external CLI agents such as Claude Code, Aider, Codex, Cursor, Gemini, Goose, Grok, OpenHands, Qwen Code, and others. · `terminal_bench/agents/installed_agents`
- [LLMs](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/llms): model access · Chat-style model provider abstraction and provider-specific wiring. · `terminal_bench/llms`
- [Parsers](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/parsers): result parsing · Benchmark-specific output interpreters that produce structured scores and pass/fail results. · `terminal_bench/parsers`
- [Docker assets](https://github.com/harbor-framework/terminal-bench/tree/main/docker): container images · Base images, agent images, and MCP server assets used to build reproducible environments. · `docker`
### Ops & reporting
Dataset/registry, dashboard persistence, scripts, and validation gates.
- [Dataset](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/dataset): dataset metadata · Dataset packaging and metadata model used for benchmark selection and export. · `terminal_bench/dataset`
- [Registry](https://github.com/harbor-framework/terminal-bench/tree/main/terminal_bench/registry): indexing · Task and dataset registry access backed by repo-level indexing data. · `terminal_bench/registry`
- [Dashboard](https://github.com/harbor-framework/terminal-bench/tree/main/dashboard): reporting app · Persistence and browsing app for benchmark results and stored runs. · `dashboard`
- [Bash scripts](https://github.com/harbor-framework/terminal-bench/tree/main/scripts_bash): automation · Operational scripts for validation, harness execution, image handling, and task checks. · `scripts_bash`
- [Python scripts](https://github.com/harbor-framework/terminal-bench/tree/main/scripts_python): automation · Repo automation for harness execution, task export, and output aggregation. · `scripts_python`
- [CI workflows](https://github.com/harbor-framework/terminal-bench/blob/main/.github/workflows): validation gates · GitHub Actions checks that enforce task integrity, Docker sanity, and benchmark consistency. · `.github/workflows`
- [Registry file](https://github.com/harbor-framework/terminal-bench/blob/main/registry.json): index data · Repo-level source of task and dataset indexing. · `registry.json`
## Connections
- Adapters → Original tasks: generate tasks · Adapters translate external benchmarks into benchmark task directories.
- TB CLI → Dataset: selects · CLI chooses datasets and run targets from metadata.
- TB CLI → Harness: launches · CLI commands start and manage harness runs.
- Harness → Agents: instantiates · Harness creates or attaches the selected agent implementation.
- Harness → Terminal: controls · Harness drives the interactive containerized terminal environment.
- Harness → Parsers: evaluates · Harness hands run output to parsers for structured scoring.
- Agents → LLMs: uses · Agent implementations call model providers or chat abstractions when needed.
- Installed CLIs → Docker assets: built from · Installed CLI agents rely on containerized images and setup scripts.
- Terminal → Docker assets: orchestrates · Terminal runtime provisions Docker Compose environments and supporting images.
- Parsers → Registry file: indexes · Parsed results feed registry-backed benchmark organization and lookup.
- Dataset → Registry: resolved by · Dataset selection is backed by registry access.
- Dashboard → Registry file: reads · Dashboard/reporting uses indexed benchmark metadata and stored results.
- Bash scripts → CI workflows: supports · Repo scripts mirror or back workflow validations.
- Python scripts → Harness: runs · Python automation wraps harness execution and output aggregation.
- CI workflows → Original tasks: validates · CI checks task structure, references, and reproducibility constraints.
- CI workflows → Adapters: checks · Workflows validate adapter-generated task artifacts and invariants.
- Core package → TB CLI: contains · CLI lives within the core runtime package boundary.
- Core package → Harness: contains · Harness implementation is part of the core package.
- Core package → Terminal: contains · Terminal control is implemented inside the core package.
- Core package → Agents: contains · Agent abstractions and implementations live in the core package.
- Core package → LLMs: contains · LLM integration is a core package subsystem.
- Core package → Parsers: contains · Parser implementations are part of the core package.
- Core package → Dataset: contains · Dataset helpers and metadata live in the core package.
- Core package → Registry: contains · Registry client is part of the core package.