How vinuni-ai20k/k5-day9-evaluation-and-failure-analysis works
Lab 9 is an evaluation and regression-gating overlay for a Lab 8 agent, not a standalone agent product. Its main workflow builds and checks labeled evaluation sets, runs agent variants across items and seeds, grades trial outcomes, produces metrics, and compares a PR report with a baseline. Separate capabilities calibrate human-vs-judge scoring, inspect individual traces, and test the gate against mutations and report-gaming traps. Evidence is limited to the README and sampled source; unsampled implementation links are omitted unless documented.
Main entry points: eval.py Eval Runner · gate.py CI Gate · judge.py Judge Calibration · mutation.py Variant Scorecard
Components 17
Evaluation
- Eval Set Builder
src/lab9/evalset.py - Failure Map
src/config/fm_map.yaml - Eval Runner
src/lab9/eval.py - Agent Variants
src/lab9/variants.py - Variant Logic
src/lab9/variant_agent.py - Trial Graders
src/lab9/graders.py
Quality Analysis
- Statistics
src/lab9/stats.py - Judge Calibration
src/lab9/judge.py - Judge Labels
src/data/eval/judge_tone_public.csv - Trace Investigator
src/lab9/trace_case.py
Regression Gate
- Metric Specs
src/config/metric_spec.yaml - CI Gate
src/lab9/gate.py - CI Sanity Check
src/lab9/ci_check.py
Gate Testing
- Variant Scorecard
src/lab9/mutation.py - Integrity Traps
src/lab9/traps.py
Other components
- Engineer
- Lab 8 Host
Connections 25
- Engineer → Eval Set Builder: builds or checks
README.md - Eval Set Builder → Failure Map: maps failure modes
README.md - Engineer → Eval Runner: runs evaluation
README.md - Eval Runner → Eval Set Builder: loads cases
src/lab9/eval.py - Eval Runner → Agent Variants: runs shape
src/lab9/eval.py - Agent Variants → Variant Logic: creates shapes
src/lab9/variants.py - Variant Logic → Lab 8 Host: uses host
src/lab9/variant_agent.py - Eval Runner → Trial Graders: scores trials
src/lab9/eval.py - Eval Runner → Statistics: aggregates metrics
src/lab9/eval.py - Engineer → Judge Calibration: calibrates judge
README.md - Judge Calibration → Judge Labels: reads labels
src/lab9/judge.py - Judge Calibration → Statistics: computes calibration
src/lab9/judge.py - Engineer → Trace Investigator: inspects trial
README.md - Trace Investigator → Eval Runner: creates trial
src/lab9/trace_case.py - Engineer → CI Gate: checks report
README.md - CI Gate → Statistics: bootstraps differences
src/lab9/gate.py - CI Gate → Eval Runner: uses item metrics
src/lab9/gate.py - CI Gate → Metric Specs: loads thresholds
README.md - CI Sanity Check → Eval Runner: checks baseline run
src/lab9/ci_check.py - CI Sanity Check → Trial Graders: checks registrations
src/lab9/ci_check.py - Engineer → Variant Scorecard: scores variants
README.md - Variant Scorecard → Eval Runner: evaluates variants
src/lab9/mutation.py - Variant Scorecard → CI Gate: checks decisions
src/lab9/mutation.py - Engineer → Integrity Traps: tests integrity
README.md - Integrity Traps → CI Gate: runs gate checks
src/lab9/traps.py