# laude-institute/terminal-bench architecture

> Architecture diagram of the GitHub repository laude-institute/terminal-bench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/laude-institute/terminal-bench
- Repository: https://github.com/laude-institute/terminal-bench

## Overview

##### Project Type and Purpose
- Terminal-Bench is a CLI‐driven benchmark framework for evaluating AI agents in real terminal environments.
- It bundles a “dataset” of end-to-end tasks (the `tasks/` folder) plus an “execution harness” (the Python package `terminal_bench/…`) that drives Docker-based sandboxes, plugins for LLM agents, parsers, and test evaluators.

##### File Structure
- Top-Level Directories
– tasks/ → hundreds of task definitions (instructions, Dockerfiles, test scripts, oracle solutions)
– terminal_bench/ → core harness code:
· cli/ → `tb` command definitions
· harness/ → orchestration logic (spawn tasks, manage runs)
· agents/ → plugin implementations for different LLM backends
· parsers/, handlers/ → interpret agent output and test results
· utils/, db.py, config.py → shared helpers, registry client, environment model
– adapters/ → bridges to external benchmarks (USACO, swebench, etc.)
– docker/ → base Docker images, example agent containers
– dashboard/ → a Postgres‐based metrics service with migrations
– discord-bot/ → optional notification integration
– .github/workflows/ → CI pipelines for building, testing, publishing tasks and images

##### Extract Insights from README
- Core Components:
– Dataset of tasks (English instructions + test scripts + reference solutions)
– Execution harness (CLI + Docker sandbox + agent plugin architecture)
- Usage: `tb run --agent <agent> --dataset-name terminal-bench-core …`
- Registry JSON (`registry.json`) maps dataset versions for reproducibility

##### Final Advice
- Be thorough: show each major package as a component.
- Plugin extensibility and Docker-based isolation clearly.
- Don’t over-complicate: focus on logical layers (CLI → Harness → Docker → Tasks → Results → Dashboard).

## Diagram (Mermaid)

```mermaid
flowchart TD
  subgraph "User Interface"
    User["User"]:::external
    CLI["CLI Layer"]:::core
  end

  subgraph "Core Harness"
    Harness["Orchestrator"]:::core
    EnvModel["Env Model & Config"]:::core
    RegistryClient["Registry Client"]:::data
  end

  subgraph "Plugin Extensions"
    PluginAgents["Agent Plugins"]:::plugin
    PluginAdapters["Benchmark Adapters"]:::plugin
    Parsers["Parser Modules"]:::plugin
    Handlers["Handler Modules"]:::plugin
  end

  subgraph "Docker Sandbox Manager"
    DockerSubproc["Docker Subprocess Utils"]:::core
    DockerCompose["Docker Compose Manager"]:::core
    TerminalIO["Terminal I/O & Session Manager"]:::core
  end

  subgraph "Tasks & Execution"
    TaskDataset["Task Dataset"]:::container
    LLMBackends["LLM Backends"]:::plugin
  end

  subgraph "Results & Dashboard"
    ResultsAgg["Results Aggregator"]:::core
    DashboardService["Dashboard Service"]:::core
    DBClient["Database Client"]:::data
  end

  subgraph "External Services"
    ExtDocker["Docker Daemon"]:::external
    ExtLLM["LLM Provider APIs"]:::external
  end

  User --> CLI
  CLI --> Harness
  Harness --> PluginAgents
  Harness --> PluginAdapters
  Harness --> RegistryClient
  Harness --> DockerCompose
  Harness --> DockerSubproc
  Harness --> TerminalIO
  Harness --> Parsers
  Harness --> Handlers
  Harness --> TaskDataset
  Harness --> LLMBackends
  TaskDataset --> DockerCompose
  DockerCompose --> ExtDocker
  LLMBackends --> ExtLLM
  Harness --> ResultsAgg
  ResultsAgg --> DBClient
  ResultsAgg --> DashboardService
  DashboardService --> DBClient

  click CLI "https://github.com/laude-institute/terminal-bench/tree/main/terminal_bench/cli"
  click Harness "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/harness/harness.py"
  click PluginAgents "https://github.com/laude-institute/terminal-bench/tree/main/terminal_bench/agents"
  click PluginAdapters "https://github.com/laude-institute/terminal-bench/tree/main/adapters"
  click Parsers "https://github.com/laude-institute/terminal-bench/tree/main/terminal_bench/parsers"
  click Handlers "https://github.com/laude-institute/terminal-bench/tree/main/terminal_bench/handlers"
  click RegistryClient "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/registry/client.py"
  click DockerSubproc "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/utils/docker_subprocess.py"
  click DockerCompose "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/terminal/docker_compose_manager.py"
  click EnvModel "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/utils/env_model.py"
  click TaskDataset "https://github.com/laude-institute/terminal-bench/tree/main/tasks"
  click LLMBackends "https://github.com/laude-institute/terminal-bench/tree/main/terminal_bench/agents/installed_agents"
  click ResultsAgg "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/handlers/trial_handler.py"
  click DashboardService "https://github.com/laude-institute/terminal-bench/tree/main/dashboard"
  click DBClient "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/db.py"
  click TerminalIO "https://github.com/laude-institute/terminal-bench/blob/main/terminal_bench/terminal/terminal.py"

  classDef core fill:#CAE1FF,stroke:#333,stroke-width:1px
  classDef plugin fill:#B3FFB3,stroke:#333,stroke-width:1px
  classDef data fill:#FFF2B3,stroke:#333,stroke-width:1px
  classDef external fill:#FFD2A6,stroke:#333,stroke-width:1px
  classDef container fill:#E0E0E0,stroke:#333,stroke-width:1px
```
