# pinchbench/skill architecture

> Architecture diagram of the GitHub repository pinchbench/skill, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pinchbench/skill
- Repository: https://github.com/pinchbench/skill

## Overview

PinchBench is a benchmark runner for evaluating models acting as OpenClaw agents on practical, multi-step tasks. The primary workflow selects tasks, loads their prompts and grading criteria, runs each task through an agent, grades its transcript and workspace, then saves results and optionally uploads them to the leaderboard. It also supports direct-API or agent-session judging, run-trend analysis, and Axiom telemetry. The tree and excerpts establish these subsystems, but do not expose the full runner wiring; uncertain internal calls are omitted.

Main entry points: [`scripts/benchmark.py`](https://github.com/pinchbench/skill/blob/main/scripts/benchmark.py) (Benchmark runner), [`scripts/lib_grading.py`](https://github.com/pinchbench/skill/blob/main/scripts/lib_grading.py) (Grading engine), [`scripts/lib_tasks.py`](https://github.com/pinchbench/skill/blob/main/scripts/lib_tasks.py) (Task loader), [`scripts/lib_agent.py`](https://github.com/pinchbench/skill/blob/main/scripts/lib_agent.py) (Agent execution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_benchmark["Benchmark workflow"]
  node_runner["Benchmark runner<br/>[benchmark.py]"]
  node_launcher["CLI launcher<br/>[run.sh]"]
  node_task_catalog["Task catalog<br/>[manifest.yaml]"]
  node_task_loader["Task loader<br/>[lib_tasks.py]"]
  node_task_files["Task definitions"]
  node_agent_exec["Agent execution<br/>[lib_agent.py]"]
  node_fws["Workspace mock lifecycle<br/>[lib_fws.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_grading["Grading engine<br/>[lib_grading.py]"]
  node_automated_checks["Automated checks"]
  node_judge_rubrics["Judge rubrics"]
end

subgraph group_reporting["Results and reporting"]
  node_results["Run results<br/>[benchmark.py]"]
  node_trend["Trend analysis<br/>[lib_trend.py]"]
  node_uploader["Result uploader<br/>[lib_upload.py]"]
  node_axiom["Axiom telemetry<br/>[lib_axiom.py]"]
end

subgraph group_integrations["External integrations"]
  node_openclaw["OpenClaw"]
  node_model_api["Model API"]
  node_judge_api["Judge model API"]
  node_leaderboard["Leaderboard service"]
  node_axiom_service["Axiom"]
end

node_user(("Benchmark user"))

node_user -->|"starts run"| node_launcher
node_launcher -->|"invokes"| node_runner
node_runner -->|"loads tasks"| node_task_loader
node_task_loader -->|"reads order"| node_task_catalog
node_task_loader -->|"reads definitions"| node_task_files
node_runner -->|"runs task"| node_agent_exec
node_agent_exec -->|"dispatches agent"| node_openclaw
node_openclaw -->|"requests completion"| node_model_api
node_runner -->|"grades execution"| node_grading
node_grading -->|"reads checks"| node_automated_checks
node_grading -->|"reads rubric"| node_judge_rubrics
node_grading -.->|"requests judge session"| node_agent_exec
node_grading -.->|"requests direct judge"| node_judge_api
node_runner -->|"writes results"| node_results
node_trend -->|"reads run history"| node_results
node_runner -.->|"submits results"| node_uploader
node_uploader -.->|"uploads submission"| node_leaderboard
node_runner -.->|"records run events"| node_axiom
node_axiom -.->|"sends telemetry"| node_axiom_service
node_runner -.->|"manages mock environment"| node_fws

click node_runner "https://github.com/pinchbench/skill/blob/main/scripts/benchmark.py"
click node_launcher "https://github.com/pinchbench/skill/blob/main/scripts/run.sh"
click node_task_catalog "https://github.com/pinchbench/skill/blob/main/tasks/manifest.yaml"
click node_task_loader "https://github.com/pinchbench/skill/blob/main/scripts/lib_tasks.py"
click node_task_files "https://github.com/pinchbench/skill/tree/main/tasks"
click node_agent_exec "https://github.com/pinchbench/skill/blob/main/scripts/lib_agent.py"
click node_fws "https://github.com/pinchbench/skill/blob/main/scripts/lib_fws.py"
click node_grading "https://github.com/pinchbench/skill/blob/main/scripts/lib_grading.py"
click node_automated_checks "https://github.com/pinchbench/skill/tree/main/tasks"
click node_judge_rubrics "https://github.com/pinchbench/skill/tree/main/tasks"
click node_results "https://github.com/pinchbench/skill/blob/main/scripts/benchmark.py"
click node_trend "https://github.com/pinchbench/skill/blob/main/scripts/lib_trend.py"
click node_uploader "https://github.com/pinchbench/skill/blob/main/scripts/lib_upload.py"
click node_axiom "https://github.com/pinchbench/skill/blob/main/scripts/lib_axiom.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_launcher,node_task_catalog,node_task_loader,node_task_files,node_agent_exec,node_fws,node_user toneBlue
class node_grading,node_automated_checks,node_judge_rubrics toneAmber
class node_results,node_trend,node_uploader,node_axiom toneMint
class node_openclaw,node_model_api,node_judge_api,node_leaderboard,node_axiom_service toneRose
```

## Components

### Benchmark workflow

- [Benchmark runner](https://github.com/pinchbench/skill/blob/main/scripts/benchmark.py): `scripts/benchmark.py`
- [CLI launcher](https://github.com/pinchbench/skill/blob/main/scripts/run.sh): `scripts/run.sh`
- [Task catalog](https://github.com/pinchbench/skill/blob/main/tasks/manifest.yaml): `tasks/manifest.yaml`
- [Task loader](https://github.com/pinchbench/skill/blob/main/scripts/lib_tasks.py): `scripts/lib_tasks.py`
- [Task definitions](https://github.com/pinchbench/skill/tree/main/tasks): `tasks`
- [Agent execution](https://github.com/pinchbench/skill/blob/main/scripts/lib_agent.py): `scripts/lib_agent.py`
- [Workspace mock lifecycle](https://github.com/pinchbench/skill/blob/main/scripts/lib_fws.py): `scripts/lib_fws.py`

### Evaluation

- [Grading engine](https://github.com/pinchbench/skill/blob/main/scripts/lib_grading.py): `scripts/lib_grading.py`
- [Automated checks](https://github.com/pinchbench/skill/tree/main/tasks): `tasks`
- [Judge rubrics](https://github.com/pinchbench/skill/tree/main/tasks): `tasks`

### Results and reporting

- [Run results](https://github.com/pinchbench/skill/blob/main/scripts/benchmark.py): `scripts/benchmark.py`
- [Trend analysis](https://github.com/pinchbench/skill/blob/main/scripts/lib_trend.py): `scripts/lib_trend.py`
- [Result uploader](https://github.com/pinchbench/skill/blob/main/scripts/lib_upload.py): `scripts/lib_upload.py`
- [Axiom telemetry](https://github.com/pinchbench/skill/blob/main/scripts/lib_axiom.py): `scripts/lib_axiom.py`

### External integrations

- **OpenClaw**
- **Model API**
- **Judge model API**
- **Leaderboard service**
- **Axiom**

### Other components

- **Benchmark user**

## Connections

- Benchmark user → CLI launcher: starts run
- CLI launcher → Benchmark runner: invokes
- Benchmark runner → Task loader: loads tasks
- Task loader → Task catalog: reads order
- Task loader → Task definitions: reads definitions
- Benchmark runner → Agent execution: runs task
- Agent execution → OpenClaw: dispatches agent
- OpenClaw → Model API: requests completion
- Benchmark runner → Grading engine: grades execution
- Grading engine → Automated checks: reads checks
- Grading engine → Judge rubrics: reads rubric
- Grading engine → Agent execution: requests judge session
- Grading engine → Judge model API: requests direct judge
- Benchmark runner → Run results: writes results
- Trend analysis → Run results: reads run history
- Benchmark runner → Result uploader: submits results
- Result uploader → Leaderboard service: uploads submission
- Benchmark runner → Axiom telemetry: records run events
- Axiom telemetry → Axiom: sends telemetry
- Benchmark runner → Workspace mock lifecycle: manages mock environment