How aws-samples/sample-code-eval-harvester works
eval-harvest turns a local repository’s merged PR review history into sealed Harbor evaluation tasks: an agent surveys review iterations, captures mechanical facts, fills candidate judgment fields, emits tasks, verifies answer absence, and assembles a dataset. The separate optional eval/ harness runs agents against the CLI and grades datapoint quality; harvest_env/ supplies its isolated Harbor execution environment. The README documents the workflow and the source index supports several internal links; unsampled implementation details are not assumed.
Main entry points: cli.py CLI Dispatch · emit.py Task Emission · runner.py Trial Runner · candidate.py Candidate Capture
Components 25
Harvest CLI
- CLI Dispatch
src/eval_harvest/cli.py - PR Survey
src/eval_harvest/survey.py - Forge Mining
src/eval_harvest/forge.py - GitLab Adapter
src/eval_harvest/gitlab.py - Candidate Capture
src/eval_harvest/candidate.py - Judgment Annotation
src/eval_harvest/annotate.py - Risk Mapping
src/eval_harvest/riskmap.py - Review Rubric
src/eval_harvest/rubric.py
Task Assembly
- Task Emission
src/eval_harvest/emit.py - Harbor Serializer
src/eval_harvest/harbor.py - TOML Writer
src/eval_harvest/tomlw.py - Task Verification
src/eval_harvest/verify.py - Answer-Absence Seal
src/eval_harvest/seal.py - Dataset Assembly
src/eval_harvest/dataset.py
Eval Harness
- Trial Runner
eval/runner.py - Trajectory Models
eval/models.py - Quality Judge
eval/judge/judge.py - Model Client
eval/judge/client.py - Judge Alignment
eval/alignment/measure.py - Eval Reporting
eval/report.py
External Runtime
- Lambda MicroVM
harvest_env/src/harvest_env/lambda_microvms.py - Harbor Runtime
- LLM Endpoint
Other components
- Coding Agent
- Local Clone
Connections 29
- Coding Agent → CLI Dispatch: drives workflow
README.md - Local Clone → PR Survey: supplies history
README.md - CLI Dispatch → PR Survey: dispatches
src/eval_harvest/cli.py - PR Survey → Forge Mining: finds PRs
src/eval_harvest/survey.py - Forge Mining → GitLab Adapter: uses adapter
src/eval_harvest/gitlab.py - CLI Dispatch → Candidate Capture: dispatches
src/eval_harvest/cli.py - Local Clone → Candidate Capture: supplies revisions
README.md - Coding Agent → Judgment Annotation: fills judgments
README.md - Judgment Annotation → Candidate Capture: updates candidate
src/eval_harvest/annotate.py - CLI Dispatch → Task Emission: dispatches
src/eval_harvest/cli.py - Candidate Capture → Task Emission: provides candidate
README.md - Task Emission → Risk Mapping: uses risk data
src/eval_harvest/emit.py - Task Emission → Review Rubric: includes rubric
README.md - Task Emission → Harbor Serializer: builds task
src/eval_harvest/emit.py - Harbor Serializer → TOML Writer: serializes values
src/eval_harvest/harbor.py - Task Emission → Task Verification: verifies task
src/eval_harvest/emit.py - Task Verification → Answer-Absence Seal: checks answer absence
src/eval_harvest/verify.py - CLI Dispatch → Task Verification: dispatches
src/eval_harvest/cli.py - CLI Dispatch → Dataset Assembly: dispatches
src/eval_harvest/cli.py - Dataset Assembly → Harbor Serializer: assembles manifest
src/eval_harvest/dataset.py - Coding Agent → Trial Runner: initiates optional eval
README.md - Trial Runner → Harbor Runtime: runs trials
eval/runner.py - Trial Runner → Lambda MicroVM: selects environment
eval/runner.py - Trial Runner → Trajectory Models: normalizes traces
eval/runner.py - Quality Judge → Model Client: requests completion
eval/judge/judge.py - Model Client → LLM Endpoint: sends prompt
- Judge Alignment → Quality Judge: measures agreement
eval/alignment/measure.py - Eval Reporting → Quality Judge: uses verdicts
eval/report.py - Trial Runner → Eval Reporting: produces results
eval/run.py