# vinuni-ai20k/k5-day9-evaluation-and-failure-analysis architecture

> Architecture diagram of the GitHub repository vinuni-ai20k/k5-day9-evaluation-and-failure-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis
- Repository: https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis

## Overview

Lab 9 is an evaluation and regression-gating overlay for a Lab 8 agent, not a standalone agent product. Its main workflow builds and checks labeled evaluation sets, runs agent variants across items and seeds, grades trial outcomes, produces metrics, and compares a PR report with a baseline. Separate capabilities calibrate human-vs-judge scoring, inspect individual traces, and test the gate against mutations and report-gaming traps. Evidence is limited to the README and sampled source; unsampled implementation links are omitted unless documented.

Main entry points: [`src/lab9/eval.py`](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/eval.py) (Eval Runner), [`src/lab9/gate.py`](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/gate.py) (CI Gate), [`src/lab9/judge.py`](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/judge.py) (Judge Calibration), [`src/lab9/mutation.py`](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/mutation.py) (Variant Scorecard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evaluation["Evaluation"]
  node_evalset["Eval Set Builder<br/>[evalset.py]"]
  node_failure_map["Failure Map<br/>[fm_map.yaml]"]
  node_eval_runner["Eval Runner<br/>[eval.py]"]
  node_variants["Agent Variants<br/>[variants.py]"]
  node_variant_agent["Variant Logic<br/>[variant_agent.py]"]
  node_graders["Trial Graders<br/>[graders.py]"]
end

subgraph group_quality["Quality Analysis"]
  node_stats["Statistics<br/>[stats.py]"]
  node_judge["Judge Calibration<br/>[judge.py]"]
  node_judge_data["Judge Labels"]
  node_trace_case["Trace Investigator<br/>[trace_case.py]"]
end

subgraph group_gate["Regression Gate"]
  node_metric_spec["Metric Specs<br/>[metric_spec.yaml]"]
  node_gate_runner["CI Gate<br/>[gate.py]"]
  node_ci_check["CI Sanity Check<br/>[ci_check.py]"]
end

subgraph group_experiments["Gate Testing"]
  node_mutation["Variant Scorecard<br/>[mutation.py]"]
  node_traps["Integrity Traps<br/>[traps.py]"]
end

node_engineer(("Engineer"))
node_lab8_host["Lab 8 Host"]

node_engineer -->|"builds or checks"| node_evalset
node_evalset -->|"maps failure modes"| node_failure_map
node_engineer -->|"runs evaluation"| node_eval_runner
node_eval_runner -->|"loads cases"| node_evalset
node_eval_runner -->|"runs shape"| node_variants
node_variants -->|"creates shapes"| node_variant_agent
node_variant_agent -->|"uses host"| node_lab8_host
node_eval_runner -->|"scores trials"| node_graders
node_eval_runner -->|"aggregates metrics"| node_stats
node_engineer -->|"calibrates judge"| node_judge
node_judge -->|"reads labels"| node_judge_data
node_judge -->|"computes calibration"| node_stats
node_engineer -->|"inspects trial"| node_trace_case
node_trace_case -->|"creates trial"| node_eval_runner
node_engineer -->|"checks report"| node_gate_runner
node_gate_runner -->|"bootstraps differences"| node_stats
node_gate_runner -->|"uses item metrics"| node_eval_runner
node_gate_runner -->|"loads thresholds"| node_metric_spec
node_ci_check -->|"checks baseline run"| node_eval_runner
node_ci_check -->|"checks registrations"| node_graders
node_engineer -->|"scores variants"| node_mutation
node_mutation -->|"evaluates variants"| node_eval_runner
node_mutation -->|"checks decisions"| node_gate_runner
node_engineer -->|"tests integrity"| node_traps
node_traps -->|"runs gate checks"| node_gate_runner

click node_evalset "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/evalset.py"
click node_failure_map "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/config/fm_map.yaml"
click node_metric_spec "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/config/metric_spec.yaml"
click node_eval_runner "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/eval.py"
click node_variants "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/variants.py"
click node_variant_agent "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/variant_agent.py"
click node_graders "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/graders.py"
click node_stats "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/stats.py"
click node_judge "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/judge.py"
click node_judge_data "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/data/eval/judge_tone_public.csv"
click node_trace_case "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/trace_case.py"
click node_gate_runner "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/gate.py"
click node_ci_check "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/ci_check.py"
click node_mutation "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/mutation.py"
click node_traps "https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/traps.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_evalset,node_failure_map,node_eval_runner,node_variants,node_variant_agent,node_graders toneBlue
class node_stats,node_judge,node_judge_data,node_trace_case toneAmber
class node_metric_spec,node_gate_runner,node_ci_check toneMint
class node_mutation,node_traps toneRose
class node_engineer,node_lab8_host toneIndigo
```

## Components

### Evaluation

- [Eval Set Builder](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/evalset.py): `src/lab9/evalset.py`
- [Failure Map](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/config/fm_map.yaml): `src/config/fm_map.yaml`
- [Eval Runner](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/eval.py): `src/lab9/eval.py`
- [Agent Variants](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/variants.py): `src/lab9/variants.py`
- [Variant Logic](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/variant_agent.py): `src/lab9/variant_agent.py`
- [Trial Graders](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/graders.py): `src/lab9/graders.py`

### Quality Analysis

- [Statistics](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/stats.py): `src/lab9/stats.py`
- [Judge Calibration](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/judge.py): `src/lab9/judge.py`
- [Judge Labels](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/data/eval/judge_tone_public.csv): `src/data/eval/judge_tone_public.csv`
- [Trace Investigator](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/trace_case.py): `src/lab9/trace_case.py`

### Regression Gate

- [Metric Specs](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/config/metric_spec.yaml): `src/config/metric_spec.yaml`
- [CI Gate](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/gate.py): `src/lab9/gate.py`
- [CI Sanity Check](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/ci_check.py): `src/lab9/ci_check.py`

### Gate Testing

- [Variant Scorecard](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/mutation.py): `src/lab9/mutation.py`
- [Integrity Traps](https://github.com/vinuni-ai20k/k5-day9-evaluation-and-failure-analysis/blob/main/src/lab9/traps.py): `src/lab9/traps.py`

### Other components

- **Engineer**
- **Lab 8 Host**

## Connections

- Engineer → Eval Set Builder: builds or checks (evidence: `README.md`)
- Eval Set Builder → Failure Map: maps failure modes (evidence: `README.md`)
- Engineer → Eval Runner: runs evaluation (evidence: `README.md`)
- Eval Runner → Eval Set Builder: loads cases (evidence: `src/lab9/eval.py`)
- Eval Runner → Agent Variants: runs shape (evidence: `src/lab9/eval.py`)
- Agent Variants → Variant Logic: creates shapes (evidence: `src/lab9/variants.py`)
- Variant Logic → Lab 8 Host: uses host (evidence: `src/lab9/variant_agent.py`)
- Eval Runner → Trial Graders: scores trials (evidence: `src/lab9/eval.py`)
- Eval Runner → Statistics: aggregates metrics (evidence: `src/lab9/eval.py`)
- Engineer → Judge Calibration: calibrates judge (evidence: `README.md`)
- Judge Calibration → Judge Labels: reads labels (evidence: `src/lab9/judge.py`)
- Judge Calibration → Statistics: computes calibration (evidence: `src/lab9/judge.py`)
- Engineer → Trace Investigator: inspects trial (evidence: `README.md`)
- Trace Investigator → Eval Runner: creates trial (evidence: `src/lab9/trace_case.py`)
- Engineer → CI Gate: checks report (evidence: `README.md`)
- CI Gate → Statistics: bootstraps differences (evidence: `src/lab9/gate.py`)
- CI Gate → Eval Runner: uses item metrics (evidence: `src/lab9/gate.py`)
- CI Gate → Metric Specs: loads thresholds (evidence: `README.md`)
- CI Sanity Check → Eval Runner: checks baseline run (evidence: `src/lab9/ci_check.py`)
- CI Sanity Check → Trial Graders: checks registrations (evidence: `src/lab9/ci_check.py`)
- Engineer → Variant Scorecard: scores variants (evidence: `README.md`)
- Variant Scorecard → Eval Runner: evaluates variants (evidence: `src/lab9/mutation.py`)
- Variant Scorecard → CI Gate: checks decisions (evidence: `src/lab9/mutation.py`)
- Engineer → Integrity Traps: tests integrity (evidence: `README.md`)
- Integrity Traps → CI Gate: runs gate checks (evidence: `src/lab9/traps.py`)
