# tiger-ai-lab/clawbench architecture

> Architecture diagram of the GitHub repository tiger-ai-lab/clawbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tiger-ai-lab/clawbench
- Repository: https://github.com/tiger-ai-lab/clawbench

## Overview

ClawBench is a framework for evaluating AI web agents on real-world online tasks. The main workflow selects tasks and models, runs an agent in a container with a browser runtime, captures browser actions and intercepted requests, then verifies task outcomes and records results for analysis. Separate evaluation utilities rescore and reproduce runs or adapt tasks for Harbor, EdgeBench, and Polar rollouts. The graph reflects the supplied tree, README description, and sampled excerpts; several subsystem connections are inferred only where entry-point names and excerpts support them, so uncertain runtime wiring is omitted.

Main entry points: [`src/clawbench/runner/run.py`](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run.py) (Single-run CLI), [`src/clawbench/runner/run_support/results.py`](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/results.py) (Run artifacts), [`src/clawbench/runner/run_support/docker.py`](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/docker.py) (Container execution), [`test-cases`](https://github.com/tiger-ai-lab/clawbench/tree/main/test-cases) (Benchmark cases)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_run["Run orchestration"]
  node_tui["Interactive TUI<br/>[tui.py]"]
  node_runner["Single-run CLI<br/>[run.py]"]
  node_batch["Batch runner<br/>[batch.py]"]
  node_config["Run configuration<br/>[config.py]"]
  node_task["Task preparation<br/>[task.py]"]
  node_containers["Container execution<br/>[docker.py]"]
  node_cases["Benchmark cases"]
end

subgraph group_runtime["Agent runtime"]
  node_browser["Browser providers<br/>[providers.py]"]
  node_harness["Agent harnesses"]
  node_interceptor["Browser interception<br/>[server.py]"]
  node_runtime_data["Run artifacts<br/>[results.py]"]
end

subgraph group_judge["Verification"]
  node_verify["Task verification<br/>[verify.py]"]
  node_llm_judge["LLM judge<br/>[judge_llm.py]"]
end

subgraph group_analysis["Evaluation tools"]
  node_analyze["Result analysis<br/>[analyze.py]"]
  node_rescore["Rescore runs<br/>[rescore.py]"]
  node_reproduce["Reproduce runs<br/>[reproduce.py]"]
end

subgraph group_integrations["External adapters"]
  node_harbor["Harbor adapter<br/>[harbor_adapter.py]"]
  node_edgebench["EdgeBench adapter"]
  node_prorl["Polar submission<br/>[submit.py]"]
end

node_user(("Benchmark user"))
node_model_api{{"Model API"}}
node_browser_service{{"Browser service"}}
node_rollout{{"Polar rollout server"}}

node_user -->|"selects run"| node_tui
node_tui -->|"launches"| node_runner
node_user -->|"starts run"| node_runner
node_user -->|"starts batch"| node_batch
node_batch -->|"runs tasks"| node_runner
node_runner -->|"loads settings"| node_config
node_runner -->|"prepares task"| node_task
node_task -->|"reads cases"| node_cases
node_runner -->|"starts container"| node_containers
node_containers -->|"runs harness"| node_harness
node_containers -->|"starts browser"| node_browser
node_browser -.->|"creates session"| node_browser_service
node_harness -.->|"requests completions"| node_model_api
node_interceptor -->|"logs requests"| node_runtime_data
node_containers -->|"copies artifacts"| node_runtime_data
node_verify -->|"writes rewards"| node_runtime_data
node_verify -->|"judges request"| node_llm_judge
node_llm_judge -.->|"queries judge"| node_model_api
node_analyze -->|"reads results"| node_runtime_data
node_rescore -->|"reads runs"| node_runtime_data
node_reproduce -->|"replays run"| node_runner
node_harbor -->|"converts tasks"| node_cases
node_prorl -->|"stages task"| node_harbor
node_prorl -.->|"submits and polls"| node_rollout
node_edgebench -->|"adapts cases"| node_cases

click node_tui "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/tui.py"
click node_runner "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run.py"
click node_batch "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/batch.py"
click node_config "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/config.py"
click node_task "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/task.py"
click node_containers "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/docker.py"
click node_cases "https://github.com/tiger-ai-lab/clawbench/tree/main/test-cases"
click node_browser "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/browser_runtime/providers.py"
click node_harness "https://github.com/tiger-ai-lab/clawbench/tree/main/src/clawbench/runtime/harnesses"
click node_interceptor "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runtime/runtime-server/server.py"
click node_runtime_data "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/results.py"
click node_verify "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runtime/harbor/verify.py"
click node_llm_judge "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/judge_llm.py"
click node_analyze "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/analyze.py"
click node_rescore "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/rescore.py"
click node_reproduce "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/reproduce.py"
click node_harbor "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/harbor_adapter.py"
click node_edgebench "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/edgebench_adapter.py"
click node_prorl "https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/prorl/submit.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tui,node_runner,node_batch,node_config,node_task,node_containers,node_cases,node_user,node_browser_service toneBlue
class node_browser,node_harness,node_interceptor,node_runtime_data toneAmber
class node_verify,node_llm_judge,node_model_api,node_rollout toneMint
class node_analyze,node_rescore,node_reproduce toneRose
class node_harbor,node_edgebench,node_prorl toneIndigo
```

## Components

### Run orchestration

- [Interactive TUI](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/tui.py): `src/clawbench/tui.py`
- [Single-run CLI](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run.py): `src/clawbench/runner/run.py`
- [Batch runner](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/batch.py): `src/clawbench/runner/batch.py`
- [Run configuration](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/config.py): `src/clawbench/runner/run_support/config.py`
- [Task preparation](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/task.py): `src/clawbench/runner/run_support/task.py`
- [Container execution](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/docker.py): `src/clawbench/runner/run_support/docker.py`
- [Benchmark cases](https://github.com/tiger-ai-lab/clawbench/tree/main/test-cases): `test-cases`

### Agent runtime

- [Browser providers](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/browser_runtime/providers.py): `src/clawbench/runner/run_support/browser_runtime/providers.py`
- [Agent harnesses](https://github.com/tiger-ai-lab/clawbench/tree/main/src/clawbench/runtime/harnesses): `src/clawbench/runtime/harnesses`
- [Browser interception](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runtime/runtime-server/server.py): `src/clawbench/runtime/runtime-server/server.py`
- [Run artifacts](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/run_support/results.py): `src/clawbench/runner/run_support/results.py`

### Verification

- [Task verification](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runtime/harbor/verify.py): `src/clawbench/runtime/harbor/verify.py`
- [LLM judge](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/runner/judge_llm.py): `src/clawbench/runner/judge_llm.py`

### Evaluation tools

- [Result analysis](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/analyze.py): `src/clawbench/eval/analyze.py`
- [Rescore runs](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/rescore.py): `src/clawbench/eval/rescore.py`
- [Reproduce runs](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/reproduce.py): `src/clawbench/eval/reproduce.py`

### External adapters

- [Harbor adapter](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/harbor_adapter.py): `src/clawbench/eval/harbor_adapter.py`
- [EdgeBench adapter](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/eval/edgebench_adapter.py): `src/clawbench/eval/edgebench_adapter.py`
- [Polar submission](https://github.com/tiger-ai-lab/clawbench/blob/main/src/clawbench/prorl/submit.py): `src/clawbench/prorl/submit.py`

### Other components

- **Benchmark user**
- **Model API**
- **Browser service**
- **Polar rollout server**

## Connections

- Benchmark user → Interactive TUI: selects run
- Interactive TUI → Single-run CLI: launches
- Benchmark user → Single-run CLI: starts run
- Benchmark user → Batch runner: starts batch
- Batch runner → Single-run CLI: runs tasks
- Single-run CLI → Run configuration: loads settings
- Single-run CLI → Task preparation: prepares task
- Task preparation → Benchmark cases: reads cases
- Single-run CLI → Container execution: starts container
- Container execution → Agent harnesses: runs harness
- Container execution → Browser providers: starts browser
- Browser providers → Browser service: creates session
- Agent harnesses → Model API: requests completions
- Browser interception → Run artifacts: logs requests
- Container execution → Run artifacts: copies artifacts
- Task verification → Run artifacts: writes rewards
- Task verification → LLM judge: judges request
- LLM judge → Model API: queries judge
- Result analysis → Run artifacts: reads results
- Rescore runs → Run artifacts: reads runs
- Reproduce runs → Single-run CLI: replays run
- Harbor adapter → Benchmark cases: converts tasks
- Polar submission → Harbor adapter: stages task
- Polar submission → Polar rollout server: submits and polls
- EdgeBench adapter → Benchmark cases: adapts cases