How ninjahawk/livenerf works
livenerf is a longitudinal model-drift benchmark. Its main workflow schedules a frozen-panel run, assembles Inspect tasks and generated items, grades model responses, and records evaluation logs for paired statistical analysis and reporting. Separate calibration/design and instrument-validation workflows establish the panel, budget and detection power; hourly and daily runners support collection. The graph follows the shown task, scheduling, provider and analysis relationships, plus capabilities explicitly described in the README; some unsampled subsystem links are therefore documented rather than inferred.
Main entry points: suite.py Inspect task suite · schedule.py Stratified scheduler · __init__.py Window statistics · hourly.py Hourly runner
Components 21
Run collection
- Hourly runner
livenerf/hourly.py - Daily runner
livenerf/daily.py - Stratified scheduler
livenerf/schedule.py - Frozen panel
data/standard_panel.json
Benchmark tasks
- Inspect task suite
livenerf/tasks/suite.py - Item generators
livenerf/generators - Code tasks
livenerf/generators/code.py - Task graders
livenerf/scorers/checks.py - Shared prompts and seeds
livenerf/common.py
Analysis and reporting
- Inspect evaluation logs
- Window statistics
livenerf/analysis/__init__.py - Results charts
livenerf/plot.py - Pre-baseline reporting
livenerf/report_prebaseline.py - Pre-baseline plots
livenerf/plot_prebaseline.py
Design and validation
- Experimental design
livenerf/design.py - Benchmark calibration
livenerf/benchmarks - Instrument validation
livenerf/validate.py
Model and usage
- Claude Code provider
livenerf/providers/claudecode.py - Usage meter
livenerf/usage.py
Other components
- Benchmark operator
- Claude Code CLI
Connections 25
- Benchmark operator → Hourly runner: starts run
README.md - Benchmark operator → Daily runner: starts run
README.md - Hourly runner → Stratified scheduler: runs schedule
livenerf/hourly.py - Daily runner → Stratified scheduler: runs schedule
livenerf/daily.py - Stratified scheduler → Frozen panel: reads panel
livenerf/schedule.py - Stratified scheduler → Inspect task suite: dispatches tasks
livenerf/schedule.py - Inspect task suite → Item generators: builds items
livenerf/tasks/suite.py - Item generators → Code tasks: dispatches family
livenerf/generators/__init__.py - Inspect task suite → Task graders: scores responses
livenerf/tasks/suite.py - Inspect task suite → Shared prompts and seeds: uses prompt helpers
livenerf/tasks/suite.py - Claude Code provider → Claude Code CLI: invokes CLI
README.md - Inspect task suite → Claude Code provider: requests model output
- Hourly runner → Usage meter: checks budget
livenerf/hourly.py - Daily runner → Usage meter: checks budget
livenerf/daily.py - Stratified scheduler → Inspect evaluation logs: writes evaluations
- Inspect evaluation logs → Window statistics: reads samples
livenerf/analysis/__init__.py - Window statistics → Results charts: supplies statistics
livenerf/plot.py - Benchmark calibration → Experimental design: informs design
livenerf/design.py - Experimental design → Frozen panel: selects panel
livenerf/design.py - Benchmark operator → Instrument validation: runs validation
livenerf/validate.py - Instrument validation → Window statistics: reports outcomes
livenerf/validate.py - Benchmark operator → Pre-baseline reporting: requests report
README.md - Benchmark operator → Pre-baseline plots: requests plots
README.md - Pre-baseline reporting → Benchmark calibration: uses calibration
livenerf/report_prebaseline.py - Pre-baseline plots → Window statistics: uses statistics
livenerf/plot_prebaseline.py