How pinchbench/skill works
PinchBench is a benchmark runner for evaluating models acting as OpenClaw agents on practical, multi-step tasks. The primary workflow selects tasks, loads their prompts and grading criteria, runs each task through an agent, grades its transcript and workspace, then saves results and optionally uploads them to the leaderboard. It also supports direct-API or agent-session judging, run-trend analysis, and Axiom telemetry. The tree and excerpts establish these subsystems, but do not expose the full runner wiring; uncertain internal calls are omitted.
Main entry points: benchmark.py Benchmark runner · lib_grading.py Grading engine · lib_tasks.py Task loader · lib_agent.py Agent execution
Components 20
Benchmark workflow
- Benchmark runner
scripts/benchmark.py - CLI launcher
scripts/run.sh - Task catalog
tasks/manifest.yaml - Task loader
scripts/lib_tasks.py - Task definitions
tasks - Agent execution
scripts/lib_agent.py - Workspace mock lifecycle
scripts/lib_fws.py
Evaluation
- Grading engine
scripts/lib_grading.py - Automated checks
tasks - Judge rubrics
tasks
Results and reporting
- Run results
scripts/benchmark.py - Trend analysis
scripts/lib_trend.py - Result uploader
scripts/lib_upload.py - Axiom telemetry
scripts/lib_axiom.py
External integrations
- OpenClaw
- Model API
- Judge model API
- Leaderboard service
- Axiom
Other components
- Benchmark user
Connections 20
- Benchmark user → CLI launcher: starts run
- CLI launcher → Benchmark runner: invokes
- Benchmark runner → Task loader: loads tasks
- Task loader → Task catalog: reads order
- Task loader → Task definitions: reads definitions
- Benchmark runner → Agent execution: runs task
- Agent execution → OpenClaw: dispatches agent
- OpenClaw → Model API: requests completion
- Benchmark runner → Grading engine: grades execution
- Grading engine → Automated checks: reads checks
- Grading engine → Judge rubrics: reads rubric
- Grading engine → Agent execution: requests judge session
- Grading engine → Judge model API: requests direct judge
- Benchmark runner → Run results: writes results
- Trend analysis → Run results: reads run history
- Benchmark runner → Result uploader: submits results
- Result uploader → Leaderboard service: uploads submission
- Benchmark runner → Axiom telemetry: records run events
- Axiom telemetry → Axiom: sends telemetry
- Benchmark runner → Workspace mock lifecycle: manages mock environment