# garrytan/g-harness-evals architecture

> Architecture diagram of the GitHub repository garrytan/g-harness-evals, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/garrytan/g-harness-evals
- Repository: https://github.com/garrytan/g-harness-evals

## Overview

harness-evals is a research harness for comparing agent configurations on fixed task sets and models. Its principal workflow plans and runs trials, records their evidence, then grades and compares results such as pass rate, cost, and time. Separate capabilities build and install gbrain memory, audit experimental outcomes, and protect or package held-out results for review and replication. The excerpts show parts of runner validation, gbrain construction, and bundle handling; unsampled analysis and custody modules are included where their role is documented or indicated by the source index.

Main entry points: [`runner`](https://github.com/garrytan/g-harness-evals/tree/main/runner) (Trial records), [`runner/run.ts`](https://github.com/garrytan/g-harness-evals/blob/main/runner/run.ts) (Trial runner), [`scripts/seal-heldout.ts`](https://github.com/garrytan/g-harness-evals/blob/main/scripts/seal-heldout.ts) (Held-out sealing), [`scripts/bundle.ts`](https://github.com/garrytan/g-harness-evals/blob/main/scripts/bundle.ts) (Bundle builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_execution["Trial execution"]
  node_runner["Trial runner<br/>[run.ts]"]
  node_contracts["Trial contracts<br/>[contracts.ts]"]
  node_baseline["Baseline endpoints"]
  node_trialdata[("Trial records")]
end

subgraph group_brain["Gbrain memory"]
  node_brainbuild["Brain builder<br/>[build-brain.ts]"]
  node_braininstall["Harness installer<br/>[build-install.ts]"]
  node_rebind["Source rebinding<br/>[rebind-sources.ts]"]
end

subgraph group_analysis["Analysis and audit"]
  node_brainusage["Brain usage<br/>[gbrain-usage.ts]"]
  node_brainladder["Brain ladder<br/>[gbrain-ladder.ts]"]
  node_followed["Instruction adherence"]
  node_followedh10["Adherence H10"]
  node_recallaudit["Recall audit<br/>[recall-audit.ts]"]
  node_compare["Run comparison<br/>[compare-run.ts]"]
  node_summarize["Run summary<br/>[summarize-run.ts]"]
  node_taxonomy["Task taxonomy<br/>[taxonomy.ts]"]
  node_matrix["Matrix analysis<br/>[matrix-analysis.ts]"]
  node_paired["Paired analysis<br/>[paired-mdd.ts]"]
  node_merge["Shard merger<br/>[merge-shards.ts]"]
  node_regrade["Trial regrading<br/>[regrade.ts]"]
end

subgraph group_custody["Held-out custody"]
  node_replicate["Replication tool<br/>[replicate.ts]"]
  node_bundle["Bundle builder<br/>[bundle.ts]"]
  node_archive["Archive codec<br/>[archive.ts]"]
  node_seal["Held-out sealing<br/>[seal-heldout.ts]"]
  node_publication["Publication audit<br/>[publish-audit.ts]"]
  node_sealedcheck["Sealed publication check"]
end

node_researcher(("Researcher"))
node_tasks["Task sets"]
node_models["Models and harnesses"]

node_researcher -->|"starts trials"| node_runner
node_tasks -->|"supplies tasks"| node_runner
node_models -->|"runs configurations"| node_runner
node_runner -->|"uses contracts"| node_contracts
node_runner -->|"records evidence"| node_trialdata
node_runner -.->|"measures endpoints"| node_baseline
node_brainbuild -->|"provides brain"| node_braininstall
node_braininstall -.->|"installs for harness"| node_models
node_rebind -.->|"rebinds sources"| node_brainbuild
node_trialdata -->|"reports usage"| node_brainusage
node_brainusage -->|"feeds ladder"| node_brainladder
node_trialdata -->|"scores adherence"| node_followed
node_followed -->|"extends analysis"| node_followedh10
node_trialdata -->|"audits recall"| node_recallaudit
node_trialdata -->|"compares runs"| node_compare
node_trialdata -->|"summarizes results"| node_summarize
node_trialdata -->|"classifies tasks"| node_taxonomy
node_trialdata -->|"analyzes matrix"| node_matrix
node_trialdata -->|"tests paired effects"| node_paired
node_trialdata -->|"merges shards"| node_merge
node_trialdata -->|"regrades trials"| node_regrade
node_researcher -->|"seals held-out"| node_seal
node_trialdata -->|"packages run"| node_bundle
node_bundle -->|"encodes archive"| node_archive
node_replicate -->|"verifies bundle"| node_bundle
node_seal -->|"checks custody"| node_sealedcheck
node_seal -.->|"audits publication"| node_publication

click node_runner "https://github.com/garrytan/g-harness-evals/blob/main/runner/run.ts"
click node_contracts "https://github.com/garrytan/g-harness-evals/blob/main/runner/contracts.ts"
click node_baseline "https://github.com/garrytan/g-harness-evals/blob/main/scripts/baseline-endpoints.ts"
click node_trialdata "https://github.com/garrytan/g-harness-evals/tree/main/runner"
click node_brainbuild "https://github.com/garrytan/g-harness-evals/blob/main/gbrain/build-brain.ts"
click node_braininstall "https://github.com/garrytan/g-harness-evals/blob/main/gbrain/build-install.ts"
click node_rebind "https://github.com/garrytan/g-harness-evals/blob/main/gbrain/rebind-sources.ts"
click node_brainusage "https://github.com/garrytan/g-harness-evals/blob/main/scripts/gbrain-usage.ts"
click node_brainladder "https://github.com/garrytan/g-harness-evals/blob/main/scripts/gbrain-ladder.ts"
click node_followed "https://github.com/garrytan/g-harness-evals/blob/main/scripts/instruction-followed.ts"
click node_followedh10 "https://github.com/garrytan/g-harness-evals/blob/main/scripts/instruction-followed-h10.ts"
click node_recallaudit "https://github.com/garrytan/g-harness-evals/blob/main/scripts/recall-audit.ts"
click node_compare "https://github.com/garrytan/g-harness-evals/blob/main/scripts/compare-run.ts"
click node_summarize "https://github.com/garrytan/g-harness-evals/blob/main/scripts/summarize-run.ts"
click node_taxonomy "https://github.com/garrytan/g-harness-evals/blob/main/scripts/taxonomy.ts"
click node_matrix "https://github.com/garrytan/g-harness-evals/blob/main/scripts/matrix-analysis.ts"
click node_paired "https://github.com/garrytan/g-harness-evals/blob/main/scripts/paired-mdd.ts"
click node_merge "https://github.com/garrytan/g-harness-evals/blob/main/scripts/merge-shards.ts"
click node_regrade "https://github.com/garrytan/g-harness-evals/blob/main/scripts/regrade.ts"
click node_replicate "https://github.com/garrytan/g-harness-evals/blob/main/scripts/replicate.ts"
click node_bundle "https://github.com/garrytan/g-harness-evals/blob/main/scripts/bundle.ts"
click node_archive "https://github.com/garrytan/g-harness-evals/blob/main/runner/archive.ts"
click node_seal "https://github.com/garrytan/g-harness-evals/blob/main/scripts/seal-heldout.ts"
click node_publication "https://github.com/garrytan/g-harness-evals/blob/main/scripts/publish-audit.ts"
click node_sealedcheck "https://github.com/garrytan/g-harness-evals/blob/main/scripts/check-sealed-publication.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_contracts,node_baseline,node_trialdata toneBlue
class node_brainbuild,node_braininstall,node_rebind toneAmber
class node_brainusage,node_brainladder,node_followed,node_followedh10,node_recallaudit,node_compare,node_summarize,node_taxonomy,node_matrix,node_paired,node_merge,node_regrade toneMint
class node_replicate,node_bundle,node_archive,node_seal,node_publication,node_sealedcheck,node_tasks toneRose
class node_researcher,node_models toneIndigo
```

## Components

### Trial execution

- [Trial runner](https://github.com/garrytan/g-harness-evals/blob/main/runner/run.ts): `runner/run.ts`
- [Trial contracts](https://github.com/garrytan/g-harness-evals/blob/main/runner/contracts.ts): `runner/contracts.ts`
- [Baseline endpoints](https://github.com/garrytan/g-harness-evals/blob/main/scripts/baseline-endpoints.ts): `scripts/baseline-endpoints.ts`
- [Trial records](https://github.com/garrytan/g-harness-evals/tree/main/runner): `runner`

### Gbrain memory

- [Brain builder](https://github.com/garrytan/g-harness-evals/blob/main/gbrain/build-brain.ts): `gbrain/build-brain.ts`
- [Harness installer](https://github.com/garrytan/g-harness-evals/blob/main/gbrain/build-install.ts): `gbrain/build-install.ts`
- [Source rebinding](https://github.com/garrytan/g-harness-evals/blob/main/gbrain/rebind-sources.ts): `gbrain/rebind-sources.ts`

### Analysis and audit

- [Brain usage](https://github.com/garrytan/g-harness-evals/blob/main/scripts/gbrain-usage.ts): `scripts/gbrain-usage.ts`
- [Brain ladder](https://github.com/garrytan/g-harness-evals/blob/main/scripts/gbrain-ladder.ts): `scripts/gbrain-ladder.ts`
- [Instruction adherence](https://github.com/garrytan/g-harness-evals/blob/main/scripts/instruction-followed.ts): `scripts/instruction-followed.ts`
- [Adherence H10](https://github.com/garrytan/g-harness-evals/blob/main/scripts/instruction-followed-h10.ts): `scripts/instruction-followed-h10.ts`
- [Recall audit](https://github.com/garrytan/g-harness-evals/blob/main/scripts/recall-audit.ts): `scripts/recall-audit.ts`
- [Run comparison](https://github.com/garrytan/g-harness-evals/blob/main/scripts/compare-run.ts): `scripts/compare-run.ts`
- [Run summary](https://github.com/garrytan/g-harness-evals/blob/main/scripts/summarize-run.ts): `scripts/summarize-run.ts`
- [Task taxonomy](https://github.com/garrytan/g-harness-evals/blob/main/scripts/taxonomy.ts): `scripts/taxonomy.ts`
- [Matrix analysis](https://github.com/garrytan/g-harness-evals/blob/main/scripts/matrix-analysis.ts): `scripts/matrix-analysis.ts`
- [Paired analysis](https://github.com/garrytan/g-harness-evals/blob/main/scripts/paired-mdd.ts): `scripts/paired-mdd.ts`
- [Shard merger](https://github.com/garrytan/g-harness-evals/blob/main/scripts/merge-shards.ts): `scripts/merge-shards.ts`
- [Trial regrading](https://github.com/garrytan/g-harness-evals/blob/main/scripts/regrade.ts): `scripts/regrade.ts`

### Held-out custody

- [Replication tool](https://github.com/garrytan/g-harness-evals/blob/main/scripts/replicate.ts): `scripts/replicate.ts`
- [Bundle builder](https://github.com/garrytan/g-harness-evals/blob/main/scripts/bundle.ts): `scripts/bundle.ts`
- [Archive codec](https://github.com/garrytan/g-harness-evals/blob/main/runner/archive.ts): `runner/archive.ts`
- [Held-out sealing](https://github.com/garrytan/g-harness-evals/blob/main/scripts/seal-heldout.ts): `scripts/seal-heldout.ts`
- [Publication audit](https://github.com/garrytan/g-harness-evals/blob/main/scripts/publish-audit.ts): `scripts/publish-audit.ts`
- [Sealed publication check](https://github.com/garrytan/g-harness-evals/blob/main/scripts/check-sealed-publication.ts): `scripts/check-sealed-publication.ts`

### Other components

- **Researcher**
- **Task sets**
- **Models and harnesses**

## Connections

- Researcher → Trial runner: starts trials (evidence: `README.md`)
- Task sets → Trial runner: supplies tasks
- Models and harnesses → Trial runner: runs configurations (evidence: `README.md`)
- Trial runner → Trial contracts: uses contracts (evidence: `runner/contracts.ts`)
- Trial runner → Trial records: records evidence (evidence: `runner/contracts.ts`)
- Trial runner → Baseline endpoints: measures endpoints
- Brain builder → Harness installer: provides brain (evidence: `gbrain/build-install.ts`)
- Harness installer → Models and harnesses: installs for harness
- Source rebinding → Brain builder: rebinds sources
- Trial records → Brain usage: reports usage (evidence: `scripts/gbrain-usage.ts`)
- Brain usage → Brain ladder: feeds ladder (evidence: `scripts/gbrain-ladder.ts`)
- Trial records → Instruction adherence: scores adherence (evidence: `scripts/instruction-followed.ts`)
- Instruction adherence → Adherence H10: extends analysis (evidence: `scripts/instruction-followed-h10.ts`)
- Trial records → Recall audit: audits recall (evidence: `scripts/recall-audit.ts`)
- Trial records → Run comparison: compares runs (evidence: `scripts/compare-run.ts`)
- Trial records → Run summary: summarizes results (evidence: `scripts/summarize-run.ts`)
- Trial records → Task taxonomy: classifies tasks (evidence: `scripts/taxonomy.ts`)
- Trial records → Matrix analysis: analyzes matrix (evidence: `scripts/matrix-analysis.ts`)
- Trial records → Paired analysis: tests paired effects (evidence: `scripts/paired-mdd.ts`)
- Trial records → Shard merger: merges shards (evidence: `scripts/merge-shards.ts`)
- Trial records → Trial regrading: regrades trials (evidence: `scripts/regrade.ts`)
- Researcher → Held-out sealing: seals held-out (evidence: `README.md`)
- Trial records → Bundle builder: packages run (evidence: `scripts/bundle.ts`)
- Bundle builder → Archive codec: encodes archive (evidence: `runner/archive.ts`)
- Replication tool → Bundle builder: verifies bundle (evidence: `scripts/replicate.ts`)
- Held-out sealing → Sealed publication check: checks custody (evidence: `scripts/check-sealed-publication.ts`)
- Held-out sealing → Publication audit: audits publication