How garrytan/g-harness-evals works
harness-evals is a research harness for comparing agent configurations on fixed task sets and models. Its principal workflow plans and runs trials, records their evidence, then grades and compares results such as pass rate, cost, and time. Separate capabilities build and install gbrain memory, audit experimental outcomes, and protect or package held-out results for review and replication. The excerpts show parts of runner validation, gbrain construction, and bundle handling; unsampled analysis and custody modules are included where their role is documented or indicated by the source index.
Main entry points: runner Trial records · run.ts Trial runner · seal-heldout.ts Held-out sealing · bundle.ts Bundle builder
Components 28
Trial execution
- Trial runner
runner/run.ts - Trial contracts
runner/contracts.ts - Baseline endpoints
scripts/baseline-endpoints.ts - Trial records
runner
Gbrain memory
- Brain builder
gbrain/build-brain.ts - Harness installer
gbrain/build-install.ts - Source rebinding
gbrain/rebind-sources.ts
Analysis and audit
- Brain usage
scripts/gbrain-usage.ts - Brain ladder
scripts/gbrain-ladder.ts - Instruction adherence
scripts/instruction-followed.ts - Adherence H10
scripts/instruction-followed-h10.ts - Recall audit
scripts/recall-audit.ts - Run comparison
scripts/compare-run.ts - Run summary
scripts/summarize-run.ts - Task taxonomy
scripts/taxonomy.ts - Matrix analysis
scripts/matrix-analysis.ts - Paired analysis
scripts/paired-mdd.ts - Shard merger
scripts/merge-shards.ts - Trial regrading
scripts/regrade.ts
Held-out custody
- Replication tool
scripts/replicate.ts - Bundle builder
scripts/bundle.ts - Archive codec
runner/archive.ts - Held-out sealing
scripts/seal-heldout.ts - Publication audit
scripts/publish-audit.ts - Sealed publication check
scripts/check-sealed-publication.ts
Other components
- Researcher
- Task sets
- Models and harnesses
Connections 27
- Researcher → Trial runner: starts trials
README.md - Task sets → Trial runner: supplies tasks
- Models and harnesses → Trial runner: runs configurations
README.md - Trial runner → Trial contracts: uses contracts
runner/contracts.ts - Trial runner → Trial records: records evidence
runner/contracts.ts - Trial runner → Baseline endpoints: measures endpoints
- Brain builder → Harness installer: provides brain
gbrain/build-install.ts - Harness installer → Models and harnesses: installs for harness
- Source rebinding → Brain builder: rebinds sources
- Trial records → Brain usage: reports usage
scripts/gbrain-usage.ts - Brain usage → Brain ladder: feeds ladder
scripts/gbrain-ladder.ts - Trial records → Instruction adherence: scores adherence
scripts/instruction-followed.ts - Instruction adherence → Adherence H10: extends analysis
scripts/instruction-followed-h10.ts - Trial records → Recall audit: audits recall
scripts/recall-audit.ts - Trial records → Run comparison: compares runs
scripts/compare-run.ts - Trial records → Run summary: summarizes results
scripts/summarize-run.ts - Trial records → Task taxonomy: classifies tasks
scripts/taxonomy.ts - Trial records → Matrix analysis: analyzes matrix
scripts/matrix-analysis.ts - Trial records → Paired analysis: tests paired effects
scripts/paired-mdd.ts - Trial records → Shard merger: merges shards
scripts/merge-shards.ts - Trial records → Trial regrading: regrades trials
scripts/regrade.ts - Researcher → Held-out sealing: seals held-out
README.md - Trial records → Bundle builder: packages run
scripts/bundle.ts - Bundle builder → Archive codec: encodes archive
runner/archive.ts - Replication tool → Bundle builder: verifies bundle
scripts/replicate.ts - Held-out sealing → Sealed publication check: checks custody
scripts/check-sealed-publication.ts - Held-out sealing → Publication audit: audits publication