# aws-samples/sample-code-eval-harvester architecture

> Architecture diagram of the GitHub repository aws-samples/sample-code-eval-harvester, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/sample-code-eval-harvester
- Repository: https://github.com/aws-samples/sample-code-eval-harvester

## Overview

eval-harvest turns a local repository’s merged PR review history into sealed Harbor evaluation tasks: an agent surveys review iterations, captures mechanical facts, fills candidate judgment fields, emits tasks, verifies answer absence, and assembles a dataset. The separate optional `eval/` harness runs agents against the CLI and grades datapoint quality; `harvest_env/` supplies its isolated Harbor execution environment. The README documents the workflow and the source index supports several internal links; unsampled implementation details are not assumed.

Main entry points: [`src/eval_harvest/cli.py`](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/cli.py) (CLI Dispatch), [`src/eval_harvest/emit.py`](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/emit.py) (Task Emission), [`eval/runner.py`](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/runner.py) (Trial Runner), [`src/eval_harvest/candidate.py`](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/candidate.py) (Candidate Capture)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["Harvest CLI"]
  node_cli_entry["CLI Dispatch<br/>[cli.py]"]
  node_survey["PR Survey<br/>[survey.py]"]
  node_forge["Forge Mining<br/>[forge.py]"]
  node_gitlab["GitLab Adapter<br/>[gitlab.py]"]
  node_capture["Candidate Capture<br/>[candidate.py]"]
  node_annotation["Judgment Annotation<br/>[annotate.py]"]
  node_risk["Risk Mapping<br/>[riskmap.py]"]
  node_rubric["Review Rubric<br/>[rubric.py]"]
end

subgraph group_task["Task Assembly"]
  node_emit["Task Emission<br/>[emit.py]"]
  node_harbor["Harbor Serializer<br/>[harbor.py]"]
  node_toml["TOML Writer<br/>[tomlw.py]"]
  node_verify["Task Verification<br/>[verify.py]"]
  node_seal["Answer-Absence Seal<br/>[seal.py]"]
  node_dataset["Dataset Assembly<br/>[dataset.py]"]
end

subgraph group_runtime["Eval Harness"]
  node_eval_run["Trial Runner<br/>[runner.py]"]
  node_trajectory["Trajectory Models<br/>[models.py]"]
  node_quality["Quality Judge<br/>[judge.py]"]
  node_judge_client["Model Client<br/>[client.py]"]
  node_alignment["Judge Alignment<br/>[measure.py]"]
  node_report["Eval Reporting<br/>[report.py]"]
end

subgraph group_integrations["External Runtime"]
  node_microvm["Lambda MicroVM<br/>[lambda_microvms.py]"]
  node_harbor_service["Harbor Runtime"]
  node_model_service["LLM Endpoint"]
end

node_agent(("Coding Agent"))
node_clone[("Local Clone")]

node_agent -->|"drives workflow"| node_cli_entry
node_clone -->|"supplies history"| node_survey
node_cli_entry -->|"dispatches"| node_survey
node_survey -->|"finds PRs"| node_forge
node_forge -->|"uses adapter"| node_gitlab
node_cli_entry -->|"dispatches"| node_capture
node_clone -->|"supplies revisions"| node_capture
node_agent -->|"fills judgments"| node_annotation
node_annotation -->|"updates candidate"| node_capture
node_cli_entry -->|"dispatches"| node_emit
node_capture -->|"provides candidate"| node_emit
node_emit -->|"uses risk data"| node_risk
node_emit -->|"includes rubric"| node_rubric
node_emit -->|"builds task"| node_harbor
node_harbor -->|"serializes values"| node_toml
node_emit -->|"verifies task"| node_verify
node_verify -->|"checks answer absence"| node_seal
node_cli_entry -->|"dispatches"| node_verify
node_cli_entry -->|"dispatches"| node_dataset
node_dataset -->|"assembles manifest"| node_harbor
node_agent -.->|"initiates optional eval"| node_eval_run
node_eval_run -->|"runs trials"| node_harbor_service
node_eval_run -->|"selects environment"| node_microvm
node_eval_run -->|"normalizes traces"| node_trajectory
node_quality -->|"requests completion"| node_judge_client
node_judge_client -.->|"sends prompt"| node_model_service
node_alignment -->|"measures agreement"| node_quality
node_report -->|"uses verdicts"| node_quality
node_eval_run -->|"produces results"| node_report

click node_cli_entry "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/cli.py"
click node_survey "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/survey.py"
click node_forge "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/forge.py"
click node_gitlab "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/gitlab.py"
click node_capture "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/candidate.py"
click node_annotation "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/annotate.py"
click node_risk "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/riskmap.py"
click node_rubric "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/rubric.py"
click node_emit "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/emit.py"
click node_harbor "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/harbor.py"
click node_toml "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/tomlw.py"
click node_verify "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/verify.py"
click node_seal "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/seal.py"
click node_dataset "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/dataset.py"
click node_eval_run "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/runner.py"
click node_trajectory "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/models.py"
click node_quality "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/judge/judge.py"
click node_judge_client "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/judge/client.py"
click node_alignment "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/alignment/measure.py"
click node_report "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/report.py"
click node_microvm "https://github.com/aws-samples/sample-code-eval-harvester/blob/main/harvest_env/src/harvest_env/lambda_microvms.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_entry,node_survey,node_forge,node_gitlab,node_capture,node_annotation,node_risk,node_rubric toneBlue
class node_emit,node_harbor,node_toml,node_verify,node_seal,node_dataset,node_clone toneAmber
class node_eval_run,node_trajectory,node_quality,node_judge_client,node_alignment,node_report toneMint
class node_microvm,node_harbor_service,node_model_service toneRose
class node_agent toneIndigo
```

## Components

### Harvest CLI

- [CLI Dispatch](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/cli.py): `src/eval_harvest/cli.py`
- [PR Survey](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/survey.py): `src/eval_harvest/survey.py`
- [Forge Mining](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/forge.py): `src/eval_harvest/forge.py`
- [GitLab Adapter](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/gitlab.py): `src/eval_harvest/gitlab.py`
- [Candidate Capture](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/candidate.py): `src/eval_harvest/candidate.py`
- [Judgment Annotation](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/annotate.py): `src/eval_harvest/annotate.py`
- [Risk Mapping](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/riskmap.py): `src/eval_harvest/riskmap.py`
- [Review Rubric](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/rubric.py): `src/eval_harvest/rubric.py`

### Task Assembly

- [Task Emission](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/emit.py): `src/eval_harvest/emit.py`
- [Harbor Serializer](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/harbor.py): `src/eval_harvest/harbor.py`
- [TOML Writer](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/tomlw.py): `src/eval_harvest/tomlw.py`
- [Task Verification](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/verify.py): `src/eval_harvest/verify.py`
- [Answer-Absence Seal](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/seal.py): `src/eval_harvest/seal.py`
- [Dataset Assembly](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/src/eval_harvest/dataset.py): `src/eval_harvest/dataset.py`

### Eval Harness

- [Trial Runner](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/runner.py): `eval/runner.py`
- [Trajectory Models](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/models.py): `eval/models.py`
- [Quality Judge](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/judge/judge.py): `eval/judge/judge.py`
- [Model Client](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/judge/client.py): `eval/judge/client.py`
- [Judge Alignment](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/alignment/measure.py): `eval/alignment/measure.py`
- [Eval Reporting](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/eval/report.py): `eval/report.py`

### External Runtime

- [Lambda MicroVM](https://github.com/aws-samples/sample-code-eval-harvester/blob/main/harvest_env/src/harvest_env/lambda_microvms.py): `harvest_env/src/harvest_env/lambda_microvms.py`
- **Harbor Runtime**
- **LLM Endpoint**

### Other components

- **Coding Agent**
- **Local Clone**

## Connections

- Coding Agent → CLI Dispatch: drives workflow (evidence: `README.md`)
- Local Clone → PR Survey: supplies history (evidence: `README.md`)
- CLI Dispatch → PR Survey: dispatches (evidence: `src/eval_harvest/cli.py`)
- PR Survey → Forge Mining: finds PRs (evidence: `src/eval_harvest/survey.py`)
- Forge Mining → GitLab Adapter: uses adapter (evidence: `src/eval_harvest/gitlab.py`)
- CLI Dispatch → Candidate Capture: dispatches (evidence: `src/eval_harvest/cli.py`)
- Local Clone → Candidate Capture: supplies revisions (evidence: `README.md`)
- Coding Agent → Judgment Annotation: fills judgments (evidence: `README.md`)
- Judgment Annotation → Candidate Capture: updates candidate (evidence: `src/eval_harvest/annotate.py`)
- CLI Dispatch → Task Emission: dispatches (evidence: `src/eval_harvest/cli.py`)
- Candidate Capture → Task Emission: provides candidate (evidence: `README.md`)
- Task Emission → Risk Mapping: uses risk data (evidence: `src/eval_harvest/emit.py`)
- Task Emission → Review Rubric: includes rubric (evidence: `README.md`)
- Task Emission → Harbor Serializer: builds task (evidence: `src/eval_harvest/emit.py`)
- Harbor Serializer → TOML Writer: serializes values (evidence: `src/eval_harvest/harbor.py`)
- Task Emission → Task Verification: verifies task (evidence: `src/eval_harvest/emit.py`)
- Task Verification → Answer-Absence Seal: checks answer absence (evidence: `src/eval_harvest/verify.py`)
- CLI Dispatch → Task Verification: dispatches (evidence: `src/eval_harvest/cli.py`)
- CLI Dispatch → Dataset Assembly: dispatches (evidence: `src/eval_harvest/cli.py`)
- Dataset Assembly → Harbor Serializer: assembles manifest (evidence: `src/eval_harvest/dataset.py`)
- Coding Agent → Trial Runner: initiates optional eval (evidence: `README.md`)
- Trial Runner → Harbor Runtime: runs trials (evidence: `eval/runner.py`)
- Trial Runner → Lambda MicroVM: selects environment (evidence: `eval/runner.py`)
- Trial Runner → Trajectory Models: normalizes traces (evidence: `eval/runner.py`)
- Quality Judge → Model Client: requests completion (evidence: `eval/judge/judge.py`)
- Model Client → LLM Endpoint: sends prompt
- Judge Alignment → Quality Judge: measures agreement (evidence: `eval/alignment/measure.py`)
- Eval Reporting → Quality Judge: uses verdicts (evidence: `eval/report.py`)
- Trial Runner → Eval Reporting: produces results (evidence: `eval/run.py`)
