How tiger-ai-lab/clawbench works
ClawBench is a framework for evaluating AI web agents on real-world online tasks. The main workflow selects tasks and models, runs an agent in a container with a browser runtime, captures browser actions and intercepted requests, then verifies task outcomes and records results for analysis. Separate evaluation utilities rescore and reproduce runs or adapt tasks for Harbor, EdgeBench, and Polar rollouts. The graph reflects the supplied tree, README description, and sampled excerpts; several subsystem connections are inferred only where entry-point names and excerpts support them, so uncertain runtime wiring is omitted.
Main entry points: run.py Single-run CLI · results.py Run artifacts · docker.py Container execution · test-cases Benchmark cases
Components 23
Run orchestration
- Interactive TUI
src/clawbench/tui.py - Single-run CLI
src/clawbench/runner/run.py - Batch runner
src/clawbench/runner/batch.py - Run configuration
src/clawbench/runner/run_support/config.py - Task preparation
src/clawbench/runner/run_support/task.py - Container execution
src/clawbench/runner/run_support/docker.py - Benchmark cases
test-cases
Agent runtime
- Browser providers
src/clawbench/runner/run_support/browser_runtime/providers.py - Agent harnesses
src/clawbench/runtime/harnesses - Browser interception
src/clawbench/runtime/runtime-server/server.py - Run artifacts
src/clawbench/runner/run_support/results.py
Verification
- Task verification
src/clawbench/runtime/harbor/verify.py - LLM judge
src/clawbench/runner/judge_llm.py
Evaluation tools
- Result analysis
src/clawbench/eval/analyze.py - Rescore runs
src/clawbench/eval/rescore.py - Reproduce runs
src/clawbench/eval/reproduce.py
External adapters
- Harbor adapter
src/clawbench/eval/harbor_adapter.py - EdgeBench adapter
src/clawbench/eval/edgebench_adapter.py - Polar submission
src/clawbench/prorl/submit.py
Other components
- Benchmark user
- Model API
- Browser service
- Polar rollout server
Connections 25
- Benchmark user → Interactive TUI: selects run
- Interactive TUI → Single-run CLI: launches
- Benchmark user → Single-run CLI: starts run
- Benchmark user → Batch runner: starts batch
- Batch runner → Single-run CLI: runs tasks
- Single-run CLI → Run configuration: loads settings
- Single-run CLI → Task preparation: prepares task
- Task preparation → Benchmark cases: reads cases
- Single-run CLI → Container execution: starts container
- Container execution → Agent harnesses: runs harness
- Container execution → Browser providers: starts browser
- Browser providers → Browser service: creates session
- Agent harnesses → Model API: requests completions
- Browser interception → Run artifacts: logs requests
- Container execution → Run artifacts: copies artifacts
- Task verification → Run artifacts: writes rewards
- Task verification → LLM judge: judges request
- LLM judge → Model API: queries judge
- Result analysis → Run artifacts: reads results
- Rescore runs → Run artifacts: reads runs
- Reproduce runs → Single-run CLI: replays run
- Harbor adapter → Benchmark cases: converts tasks
- Polar submission → Harbor adapter: stages task
- Polar submission → Polar rollout server: submits and polls
- EdgeBench adapter → Benchmark cases: adapts cases