# fstandhartinger/jevbench architecture

> Architecture diagram of the GitHub repository fstandhartinger/jevbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fstandhartinger/jevbench
- Repository: https://github.com/fstandhartinger/jevbench

## Overview

JevBench evaluates typed decision models on canonical benchmark tasks and produces per-item measurements and aggregate scores. The principal workflow is: an engineer invokes the CLI with task records and an adapter; the runner reserves budget, queries the selected model, validates and scores its answer, and records measurements; summarization computes metrics for comparison. Adapters cover remote APIs and local inference, while versioned composite modules define published scoring methods. Source excerpts establish the run path; some adapter wiring and aggregate-release processing are documented but not fully sampled.

Main entry points: [`jevbench/cli.py`](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/cli.py) (CLI), [`jevbench/runner.py`](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/runner.py) (Runner), [`jevbench/summarize.py`](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/summarize.py) (Summarization), [`jevbench/adapters`](https://github.com/fstandhartinger/jevbench/tree/main/jevbench/adapters) (Model adapters)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Benchmark entry"]
  node_cli["CLI<br/>[cli.py]"]
  node_tasks["Task records<br/>[tasks.py]"]
end

subgraph group_models["Model connections"]
  node_adapter_api["Adapter contract<br/>[base.py]"]
  node_adapter_impl["Model adapters"]
  node_djev["djev adapter<br/>[djev.py]"]
  node_openai_adapter["OpenAI-compatible adapter<br/>[openai_compat.py]"]
end

subgraph group_execution["Run execution"]
  node_runner["Runner<br/>[runner.py]"]
  node_budget[("Budget ledger<br/>[budget.py]")]
end

subgraph group_analysis["Scoring and analysis"]
  node_scoring["Answer scoring<br/>[scoring.py]"]
  node_metrics["Benchmark metrics<br/>[metrics.py]"]
  node_summarize["Summarization<br/>[summarize.py]"]
  node_composite_v13["v1.3 composite<br/>[composite_v13.py]"]
  node_composite_v14["v1.4 composite<br/>[composite_v14.py]"]
end

node_engineer(("Engineer"))
node_model_service{{"Model services"}}

node_engineer -->|"invokes"| node_cli
node_cli -->|"loads"| node_tasks
node_cli -->|"selects"| node_adapter_impl
node_cli -->|"starts run"| node_runner
node_cli -->|"starts summary"| node_summarize
node_runner -->|"calls adapter"| node_adapter_api
node_adapter_impl -->|"implements"| node_adapter_api
node_runner -->|"reserves funds"| node_budget
node_runner -->|"scores answers"| node_scoring
node_djev -->|"sends requests"| node_model_service
node_openai_adapter -->|"posts requests"| node_model_service
node_summarize -->|"uses outcomes"| node_scoring
node_summarize -->|"computes metrics"| node_metrics
node_composite_v14 -->|"reuses scoring"| node_composite_v13

click node_cli "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/cli.py"
click node_tasks "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/tasks.py"
click node_adapter_api "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/adapters/base.py"
click node_adapter_impl "https://github.com/fstandhartinger/jevbench/tree/main/jevbench/adapters"
click node_djev "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/adapters/djev.py"
click node_openai_adapter "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/adapters/openai_compat.py"
click node_runner "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/runner.py"
click node_budget "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/budget.py"
click node_scoring "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/scoring.py"
click node_metrics "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/metrics.py"
click node_summarize "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/summarize.py"
click node_composite_v13 "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/composite_v13.py"
click node_composite_v14 "https://github.com/fstandhartinger/jevbench/blob/main/jevbench/composite_v14.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_tasks toneBlue
class node_adapter_api,node_adapter_impl,node_djev,node_openai_adapter toneAmber
class node_runner,node_budget toneMint
class node_scoring,node_metrics,node_summarize,node_composite_v13,node_composite_v14 toneRose
class node_engineer,node_model_service toneIndigo
```

## Components

### Benchmark entry

- [CLI](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/cli.py): `jevbench/cli.py`
- [Task records](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/tasks.py): `jevbench/tasks.py`

### Model connections

- [Adapter contract](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/adapters/base.py): `jevbench/adapters/base.py`
- [Model adapters](https://github.com/fstandhartinger/jevbench/tree/main/jevbench/adapters): `jevbench/adapters`
- [djev adapter](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/adapters/djev.py): `jevbench/adapters/djev.py`
- [OpenAI-compatible adapter](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/adapters/openai_compat.py): `jevbench/adapters/openai_compat.py`

### Run execution

- [Runner](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/runner.py): `jevbench/runner.py`
- [Budget ledger](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/budget.py): `jevbench/budget.py`

### Scoring and analysis

- [Answer scoring](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/scoring.py): `jevbench/scoring.py`
- [Benchmark metrics](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/metrics.py): `jevbench/metrics.py`
- [Summarization](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/summarize.py): `jevbench/summarize.py`
- [v1.3 composite](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/composite_v13.py): `jevbench/composite_v13.py`
- [v1.4 composite](https://github.com/fstandhartinger/jevbench/blob/main/jevbench/composite_v14.py): `jevbench/composite_v14.py`

### Other components

- **Engineer**
- **Model services**

## Connections

- Engineer → CLI: invokes (evidence: `jevbench/cli.py`)
- CLI → Task records: loads (evidence: `jevbench/cli.py`)
- CLI → Model adapters: selects (evidence: `jevbench/cli.py`)
- CLI → Runner: starts run (evidence: `jevbench/cli.py`)
- CLI → Summarization: starts summary (evidence: `jevbench/cli.py`)
- Runner → Adapter contract: calls adapter (evidence: `jevbench/runner.py`)
- Model adapters → Adapter contract: implements (evidence: `jevbench/adapters/__init__.py`)
- Runner → Budget ledger: reserves funds (evidence: `jevbench/runner.py`)
- Runner → Answer scoring: scores answers (evidence: `jevbench/runner.py`)
- djev adapter → Model services: sends requests (evidence: `jevbench/adapters/djev.py`)
- OpenAI-compatible adapter → Model services: posts requests (evidence: `jevbench/adapters/openai_compat.py`)
- Summarization → Answer scoring: uses outcomes (evidence: `jevbench/summarize.py`)
- Summarization → Benchmark metrics: computes metrics (evidence: `jevbench/summarize.py`)
- v1.4 composite → v1.3 composite: reuses scoring (evidence: `jevbench/composite_v14.py`)
