# danielgshea/jev-as-a-judge architecture

> Architecture diagram of the GitHub repository danielgshea/jev-as-a-judge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/danielgshea/jev-as-a-judge
- Repository: https://github.com/danielgshea/jev-as-a-judge

## Overview

This repository is an experiment comparing Jev and LLM-as-a-judge on a weather agent. The main workflow runs weather requests, captures agent answers and search evidence, scores frozen outputs with multiple judges, and analyzes accuracy, repeatability, cost, and latency against human labels. The graph also includes the documented dataset/LangSmith evaluation and visualization branches. Some subsystem wiring is based on README descriptions and module boundaries because those source files were not sampled.

Main entry points: [`src/evals/offline_evals.py`](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/offline_evals.py) (Run and capture), [`src/evals/judge_reliability.py`](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judge_reliability.py) (Repeated trials), [`analysis/visualize.py`](https://github.com/danielgshea/jev-as-a-judge/blob/main/analysis/visualize.py) (Benchmark charts), [`src/evals/judges/system_one.py`](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/system_one.py) (Jev evaluators)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_weather["Weather agent"]
  node_weather_agent["Weather agent<br/>[agent.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_target["Run and capture<br/>[offline_evals.py]"]
  node_dataset[("Evaluation cases<br/>[dataset.py]")]
  node_judge_state["Judge state<br/>[__init__.py]"]
  node_jev_judges["Jev evaluators<br/>[system_one.py]"]
  node_llm_judges["LLM evaluators<br/>[llm.py]"]
  node_model_config["Judge models<br/>[model.py]"]
  node_reliability["Repeated trials"]
end

subgraph group_analysis["Benchmark analysis"]
  node_oracle_labels["Oracle labels<br/>[oracle-labels.json]"]
  node_frozen_benchmark["Frozen benchmark<br/>[benchmark.json]"]
  node_archive["Archive benchmark"]
  node_analysis_common["Analysis helpers<br/>[common.py]"]
  node_trace_metrics["Trace metrics"]
  node_visualize["Benchmark charts<br/>[visualize.py]"]
end

subgraph group_services["External services"]
  node_tavily["Tavily search"]
  node_typesafe["TypeSafe Jev"]
  node_model_gateway["LLM gateway"]
  node_langsmith[("LangSmith")]
end

node_engineer(("Experiment runner"))
node_oracle_reviewer(("Human reviewer"))

node_engineer -->|"runs evaluation"| node_eval_target
node_engineer -->|"runs trials"| node_reliability
node_weather_agent -->|"searches weather"| node_tavily
node_eval_target -->|"invokes agent"| node_weather_agent
node_eval_target -->|"loads cases"| node_dataset
node_eval_target -->|"runs evaluators"| node_jev_judges
node_eval_target -->|"runs evaluators"| node_llm_judges
node_eval_target -.->|"records experiment"| node_langsmith
node_jev_judges -->|"builds state"| node_judge_state
node_llm_judges -->|"builds state"| node_judge_state
node_jev_judges -->|"classifies state"| node_typesafe
node_llm_judges -->|"creates model"| node_model_config
node_model_config -.->|"invokes model"| node_model_gateway
node_reliability -->|"reuses target"| node_eval_target
node_reliability -->|"repeats judgments"| node_jev_judges
node_reliability -->|"repeats judgments"| node_llm_judges
node_reliability -->|"loads cases"| node_dataset
node_reliability -.->|"records trials"| node_langsmith
node_oracle_reviewer -->|"labels responses"| node_oracle_labels
node_archive -->|"reads experiment"| node_langsmith
node_archive -->|"writes archive"| node_frozen_benchmark
node_archive -->|"uses summaries"| node_reliability
node_trace_metrics -->|"queries traces"| node_langsmith
node_trace_metrics -->|"uses helpers"| node_analysis_common
node_visualize -->|"loads metrics"| node_trace_metrics
node_visualize -->|"scores oracle"| node_analysis_common
node_visualize -->|"reads labels"| node_oracle_labels
node_visualize -->|"reads cases"| node_frozen_benchmark

click node_weather_agent "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/weather_agent/agent.py"
click node_eval_target "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/offline_evals.py"
click node_dataset "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/dataset.py"
click node_judge_state "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/__init__.py"
click node_jev_judges "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/system_one.py"
click node_llm_judges "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/llm.py"
click node_model_config "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/model.py"
click node_reliability "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judge_reliability.py"
click node_oracle_labels "https://github.com/danielgshea/jev-as-a-judge/blob/main/assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/oracle-labels.json"
click node_frozen_benchmark "https://github.com/danielgshea/jev-as-a-judge/blob/main/assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/benchmark.json"
click node_archive "https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/archive_benchmark.py"
click node_analysis_common "https://github.com/danielgshea/jev-as-a-judge/blob/main/analysis/common.py"
click node_trace_metrics "https://github.com/danielgshea/jev-as-a-judge/blob/main/analysis/query_trace_metrics.py"
click node_visualize "https://github.com/danielgshea/jev-as-a-judge/blob/main/analysis/visualize.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_weather_agent,node_oracle_reviewer toneBlue
class node_eval_target,node_dataset,node_judge_state,node_jev_judges,node_llm_judges,node_model_config,node_reliability toneAmber
class node_oracle_labels,node_frozen_benchmark,node_archive,node_analysis_common,node_trace_metrics,node_visualize toneMint
class node_tavily,node_typesafe,node_model_gateway,node_langsmith toneRose
class node_engineer toneIndigo
```

## Components

### Weather agent

- [Weather agent](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/weather_agent/agent.py): `src/weather_agent/agent.py`

### Evaluation

- [Run and capture](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/offline_evals.py): `src/evals/offline_evals.py`
- [Evaluation cases](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/dataset.py): `src/evals/dataset.py`
- [Judge state](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/__init__.py): `src/evals/judges/__init__.py`
- [Jev evaluators](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/system_one.py): `src/evals/judges/system_one.py`
- [LLM evaluators](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judges/llm.py): `src/evals/judges/llm.py`
- [Judge models](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/model.py): `src/evals/model.py`
- [Repeated trials](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/judge_reliability.py): `src/evals/judge_reliability.py`

### Benchmark analysis

- [Oracle labels](https://github.com/danielgshea/jev-as-a-judge/blob/main/assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/oracle-labels.json): `assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/oracle-labels.json`
- [Frozen benchmark](https://github.com/danielgshea/jev-as-a-judge/blob/main/assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/benchmark.json): `assets/benchmark-jev-luna-terra-sonnet/6d08df72-c878-458c-b7c5-a7824ee6e721/benchmark.json`
- [Archive benchmark](https://github.com/danielgshea/jev-as-a-judge/blob/main/src/evals/archive_benchmark.py): `src/evals/archive_benchmark.py`
- [Analysis helpers](https://github.com/danielgshea/jev-as-a-judge/blob/main/analysis/common.py): `analysis/common.py`
- [Trace metrics](https://github.com/danielgshea/jev-as-a-judge/blob/main/analysis/query_trace_metrics.py): `analysis/query_trace_metrics.py`
- [Benchmark charts](https://github.com/danielgshea/jev-as-a-judge/blob/main/analysis/visualize.py): `analysis/visualize.py`

### External services

- **Tavily search**
- **TypeSafe Jev**
- **LLM gateway**
- **LangSmith**

### Other components

- **Experiment runner**
- **Human reviewer**

## Connections

- Experiment runner → Run and capture: runs evaluation
- Experiment runner → Repeated trials: runs trials
- Weather agent → Tavily search: searches weather
- Run and capture → Weather agent: invokes agent
- Run and capture → Evaluation cases: loads cases
- Run and capture → Jev evaluators: runs evaluators
- Run and capture → LLM evaluators: runs evaluators
- Run and capture → LangSmith: records experiment
- Jev evaluators → Judge state: builds state
- LLM evaluators → Judge state: builds state
- Jev evaluators → TypeSafe Jev: classifies state
- LLM evaluators → Judge models: creates model
- Judge models → LLM gateway: invokes model
- Repeated trials → Run and capture: reuses target
- Repeated trials → Jev evaluators: repeats judgments
- Repeated trials → LLM evaluators: repeats judgments
- Repeated trials → Evaluation cases: loads cases
- Repeated trials → LangSmith: records trials
- Human reviewer → Oracle labels: labels responses
- Archive benchmark → LangSmith: reads experiment
- Archive benchmark → Frozen benchmark: writes archive
- Archive benchmark → Repeated trials: uses summaries
- Trace metrics → LangSmith: queries traces
- Trace metrics → Analysis helpers: uses helpers
- Benchmark charts → Trace metrics: loads metrics
- Benchmark charts → Analysis helpers: scores oracle
- Benchmark charts → Oracle labels: reads labels
- Benchmark charts → Frozen benchmark: reads cases