How laude-institute/terminal-bench works
Project Type and Purpose
• Terminal-Bench is a CLI‐driven benchmark framework for evaluating AI agents in real terminal environments.
• It bundles a “dataset” of end-to-end tasks (the tasks/ folder) plus an “execution harness” (the Python package terminal_bench/…) that drives Docker-based sandboxes, plugins for LLM agents, parsers, and test evaluators.
File Structure
• Top-Level Directories
– tasks/ → hundreds of task definitions (instructions, Dockerfiles, test scripts, oracle solutions)
– terminal_bench/ → core harness code:
· cli/ → tb command definitions
· harness/ → orchestration logic (spawn tasks, manage runs)
· agents/ → plugin implementations for different LLM backends
· parsers/, handlers/ → interpret agent output and test results
· utils/, db.py, config.py → shared helpers, registry client, environment model
– adapters/ → bridges to external benchmarks (USACO, swebench, etc.)
– docker/ → base Docker images, example agent containers
– dashboard/ → a Postgres‐based metrics service with migrations
– discord-bot/ → optional notification integration
– .github/workflows/ → CI pipelines for building, testing, publishing tasks and images
Extract Insights from README
• Core Components:
– Dataset of tasks (English instructions + test scripts + reference solutions)
– Execution harness (CLI + Docker sandbox + agent plugin architecture)
• Usage: tb run --agent <agent> --dataset-name terminal-bench-core …
• Registry JSON (registry.json) maps dataset versions for reproducibility
Final Advice
• Be thorough: show each major package as a component.
• Plugin extensibility and Docker-based isolation clearly.
• Don’t over-complicate: focus on logical layers (CLI → Harness → Docker → Tasks → Results → Dashboard).