How fstandhartinger/jevbench works
JevBench evaluates typed decision models on canonical benchmark tasks and produces per-item measurements and aggregate scores. The principal workflow is: an engineer invokes the CLI with task records and an adapter; the runner reserves budget, queries the selected model, validates and scores its answer, and records measurements; summarization computes metrics for comparison. Adapters cover remote APIs and local inference, while versioned composite modules define published scoring methods. Source excerpts establish the run path; some adapter wiring and aggregate-release processing are documented but not fully sampled.
Main entry points: cli.py CLI · runner.py Runner · summarize.py Summarization · adapters Model adapters
Components 15
Benchmark entry
- CLI
jevbench/cli.py - Task records
jevbench/tasks.py
Model connections
- Adapter contract
jevbench/adapters/base.py - Model adapters
jevbench/adapters - djev adapter
jevbench/adapters/djev.py - OpenAI-compatible adapter
jevbench/adapters/openai_compat.py
Run execution
- Runner
jevbench/runner.py - Budget ledger
jevbench/budget.py
Scoring and analysis
- Answer scoring
jevbench/scoring.py - Benchmark metrics
jevbench/metrics.py - Summarization
jevbench/summarize.py - v1.3 composite
jevbench/composite_v13.py - v1.4 composite
jevbench/composite_v14.py
Other components
- Engineer
- Model services
Connections 14
- Engineer → CLI: invokes
jevbench/cli.py - CLI → Task records: loads
jevbench/cli.py - CLI → Model adapters: selects
jevbench/cli.py - CLI → Runner: starts run
jevbench/cli.py - CLI → Summarization: starts summary
jevbench/cli.py - Runner → Adapter contract: calls adapter
jevbench/runner.py - Model adapters → Adapter contract: implements
jevbench/adapters/__init__.py - Runner → Budget ledger: reserves funds
jevbench/runner.py - Runner → Answer scoring: scores answers
jevbench/runner.py - djev adapter → Model services: sends requests
jevbench/adapters/djev.py - OpenAI-compatible adapter → Model services: posts requests
jevbench/adapters/openai_compat.py - Summarization → Answer scoring: uses outcomes
jevbench/summarize.py - Summarization → Benchmark metrics: computes metrics
jevbench/summarize.py - v1.4 composite → v1.3 composite: reuses scoring
jevbench/composite_v14.py