How tencent/workbuddy-bench works
WorkBuddy Bench is an evaluation framework for coding agents across Code, Web, Office, and Security tasks. Its principal workflow configures a model, harness, and dataset; prepares and runs task trials in Docker; optionally routes model traffic through a logging/protocol proxy; then verifies results and scores the evaluation. The graph includes the documented evaluation stages and major verifier and proxy capabilities. Some runner and post-processing internals were not excerpted, so links not supported by the README or source index are omitted.
Main entry points: sharded_eval.py Trial orchestration · pipeline.py Protocol and interceptors · resolve_manifest.py Run manifest · engine.py Composite verifier engine
Components 24
Evaluation setup
- Job and model config
src/workbuddy_bench/runner/config_loaders.py - Run manifest
src/workbuddy_bench/runner/resolve_manifest.py - Task preparation
src/workbuddy_bench/runner/prepare_tasks.py - Benchmark subsets
Trial execution
- Trial orchestration
src/workbuddy_bench/runner/sharded_eval.py - Docker sandbox
src/workbuddy_bench/runner/docker_environment.py - Agent harness adapters
src/workbuddy_bench/runner/harness_adapters.py - Agent usage capture
src/workbuddy_bench/agents/_agent_user.py
Model traffic proxy
- Proxy routes
src/workbuddy_bench/proxy/config.py - Proxy request entry
src/workbuddy_bench/proxy/main.py - Protocol and interceptors
src/workbuddy_bench/proxy/pipeline.py - Anthropic/OpenAI mapping
src/workbuddy_bench/proxy/protocols/a2o.py - Request logging
src/workbuddy_bench/proxy/interceptors/logger.py
Verification and scoring
- Judge profile dispatch
src/workbuddy_bench/judge/profiles/dispatcher.py - Composite verifier engine
src/workbuddy_bench/judge/core/engine.py - Pytest and rule judges
src/workbuddy_bench/judge/runners/harbor_pytest.py - Artifact inspection
src/workbuddy_bench/judge/evidence/artifact_inspector.py - Command execution
src/workbuddy_bench/judge/runtime/command.py - Correctness scoring
src/workbuddy_bench/scorer/scorer.py - LLM post-judge
src/workbuddy_bench/scorer/llm_judge.py - Evaluation results
Other components
- Benchmark operator
- Agent CLI
- Model backend
Connections 25
- Benchmark operator → Job and model config: configures
README.md - Job and model config → Run manifest: resolves
src/workbuddy_bench/runner/resolve_manifest.py - Run manifest → Task preparation: prepares tasks
- Benchmark subsets → Task preparation: supplies tasks
README.md - Run manifest → Trial orchestration: launches trials
- Trial orchestration → Docker sandbox: runs in
README.md - Trial orchestration → Agent harness adapters: dispatches
README.md - Agent harness adapters → Agent CLI: runs
README.md - Agent CLI → Proxy request entry: sends model requests
- Agent usage capture → Trial orchestration: captures usage
- Run manifest → Proxy routes: configures routes
- Proxy request entry → Protocol and interceptors: dispatches requests
src/workbuddy_bench/proxy/main.py - Protocol and interceptors → Proxy routes: uses routes
src/workbuddy_bench/proxy/pipeline.py - Protocol and interceptors → Anthropic/OpenAI mapping: converts protocol
src/workbuddy_bench/proxy/pipeline.py - Protocol and interceptors → Request logging: logs exchanges
src/workbuddy_bench/proxy/pipeline.py - Protocol and interceptors → Model backend: forwards requests
src/workbuddy_bench/proxy/pipeline.py - Trial orchestration → Judge profile dispatch: routes judging
- Judge profile dispatch → Composite verifier engine: dispatches profile
- Composite verifier engine → Pytest and rule judges: runs judges
- Pytest and rule judges → Command execution: executes commands
src/workbuddy_bench/judge/runners/harbor_pytest.py - Composite verifier engine → Artifact inspection: collects evidence
- Trial orchestration → Correctness scoring: scores trial
README.md - Correctness scoring → Evaluation results: writes scores
README.md - Evaluation results → LLM post-judge: post-processes
src/workbuddy_bench/scorer/llm_judge.py - LLM post-judge → Evaluation results: updates scores
src/workbuddy_bench/scorer/llm_judge.py