How synthiaresearch/goldilocks-bench-lite works
Goldilocks Bench Lite evaluates whether coding agents fix warranted issues while leaving protected code untouched. The main workflow runs an agent against a task repository, grades its change against hidden behavioral and preservation checks, then aggregates run scores into act (A) and restraint (R). The task corpus spans TinyDB, JMESPath, and boltons. The README documents these capabilities; source excerpts are limited to TinyDB storage and database code, so task-internal runtime wiring is not expanded here.
Main entry points: run.py Model runner · grade.py Change grader · adapter.py Agent adapter · score.py Score aggregator
Components 12
Agent Runs
- Model runner
scripts/run.py - Agent adapter
src/goldilocks/adapter.py
Task Corpus
- TinyDB tasks
tasks/gb-lad-d03/repo/tinydb - JMESPath tasks
tasks/jm-gb-lad-e03/repo/jmespath - boltons tasks
tasks/bo-gb-lad-d05/repo/boltons
Scoring
- Baseline policies
src/goldilocks/baselines.py - Change grader
src/goldilocks/grade.py - Score aggregator
scripts/score.py
Other components
- Benchmark user
- Model API
- Run results
- Hidden checks
Connections 12
- Benchmark user → Model runner: starts run
- Model runner → Agent adapter: invokes
- Agent adapter → Model API: requests completion
- Model runner → TinyDB tasks: selects task
- Model runner → JMESPath tasks: selects task
- Model runner → boltons tasks: selects task
- Model runner → Change grader: grades change
- Change grader → Hidden checks: evaluates against
- Baseline policies → Change grader: supplies policies
- Model runner → Run results: writes runs
- Benchmark user → Score aggregator: requests scores
- Score aggregator → Run results: reads runs