How danielgshea/jev-as-a-judge works
This repository is an experiment comparing Jev and LLM-as-a-judge on a weather agent. The main workflow runs weather requests, captures agent answers and search evidence, scores frozen outputs with multiple judges, and analyzes accuracy, repeatability, cost, and latency against human labels. The graph also includes the documented dataset/LangSmith evaluation and visualization branches. Some subsystem wiring is based on README descriptions and module boundaries because those source files were not sampled.
Main entry points: offline_evals.py Run and capture · judge_reliability.py Repeated trials · visualize.py Benchmark charts · system_one.py Jev evaluators
Components 20
Weather agent
- Weather agent
src/weather_agent/agent.py
Evaluation
- Run and capture
src/evals/offline_evals.py - Evaluation cases
src/evals/dataset.py - Judge state
src/evals/judges/__init__.py - Jev evaluators
src/evals/judges/system_one.py - LLM evaluators
src/evals/judges/llm.py - Judge models
src/evals/model.py - Repeated trials
src/evals/judge_reliability.py
Benchmark analysis
- Oracle labels
assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/oracle-labels.json - Frozen benchmark
assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/benchmark.json - Archive benchmark
src/evals/archive_benchmark.py - Analysis helpers
analysis/common.py - Trace metrics
analysis/query_trace_metrics.py - Benchmark charts
analysis/visualize.py
External services
- Tavily search
- TypeSafe Jev
- LLM gateway
- LangSmith
Other components
- Experiment runner
- Human reviewer
Connections 28
- Experiment runner → Run and capture: runs evaluation
- Experiment runner → Repeated trials: runs trials
- Weather agent → Tavily search: searches weather
- Run and capture → Weather agent: invokes agent
- Run and capture → Evaluation cases: loads cases
- Run and capture → Jev evaluators: runs evaluators
- Run and capture → LLM evaluators: runs evaluators
- Run and capture → LangSmith: records experiment
- Jev evaluators → Judge state: builds state
- LLM evaluators → Judge state: builds state
- Jev evaluators → TypeSafe Jev: classifies state
- LLM evaluators → Judge models: creates model
- Judge models → LLM gateway: invokes model
- Repeated trials → Run and capture: reuses target
- Repeated trials → Jev evaluators: repeats judgments
- Repeated trials → LLM evaluators: repeats judgments
- Repeated trials → Evaluation cases: loads cases
- Repeated trials → LangSmith: records trials
- Human reviewer → Oracle labels: labels responses
- Archive benchmark → LangSmith: reads experiment
- Archive benchmark → Frozen benchmark: writes archive
- Archive benchmark → Repeated trials: uses summaries
- Trace metrics → LangSmith: queries traces
- Trace metrics → Analysis helpers: uses helpers
- Benchmark charts → Trace metrics: loads metrics
- Benchmark charts → Analysis helpers: scores oracle
- Benchmark charts → Oracle labels: reads labels
- Benchmark charts → Frozen benchmark: reads cases