# ninjahawk/livenerf architecture

> Architecture diagram of the GitHub repository ninjahawk/livenerf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ninjahawk/livenerf
- Repository: https://github.com/ninjahawk/livenerf

## Overview

livenerf is a longitudinal model-drift benchmark. Its main workflow schedules a frozen-panel run, assembles Inspect tasks and generated items, grades model responses, and records evaluation logs for paired statistical analysis and reporting. Separate calibration/design and instrument-validation workflows establish the panel, budget and detection power; hourly and daily runners support collection. The graph follows the shown task, scheduling, provider and analysis relationships, plus capabilities explicitly described in the README; some unsampled subsystem links are therefore documented rather than inferred.

Main entry points: [`livenerf/tasks/suite.py`](https://github.com/ninjahawk/livenerf/blob/main/livenerf/tasks/suite.py) (Inspect task suite), [`livenerf/schedule.py`](https://github.com/ninjahawk/livenerf/blob/main/livenerf/schedule.py) (Stratified scheduler), [`livenerf/analysis/__init__.py`](https://github.com/ninjahawk/livenerf/blob/main/livenerf/analysis/__init__.py) (Window statistics), [`livenerf/hourly.py`](https://github.com/ninjahawk/livenerf/blob/main/livenerf/hourly.py) (Hourly runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Run collection"]
  node_hourly["Hourly runner<br/>[hourly.py]"]
  node_daily["Daily runner<br/>[daily.py]"]
  node_scheduler["Stratified scheduler<br/>[schedule.py]"]
  node_panel["Frozen panel"]
end

subgraph group_benchmark["Benchmark tasks"]
  node_tasks["Inspect task suite<br/>[suite.py]"]
  node_generators["Item generators"]
  node_codegen["Code tasks<br/>[code.py]"]
  node_scorers["Task graders<br/>[checks.py]"]
  node_common["Shared prompts and seeds<br/>[common.py]"]
end

subgraph group_measurement["Analysis and reporting"]
  node_eval_logs[("Inspect evaluation logs")]
  node_analysis["Window statistics<br/>[__init__.py]"]
  node_plot["Results charts<br/>[plot.py]"]
  node_prebaseline["Pre-baseline reporting"]
  node_preplot["Pre-baseline plots"]
end

subgraph group_design["Design and validation"]
  node_design["Experimental design<br/>[design.py]"]
  node_calibration["Benchmark calibration"]
  node_validation["Instrument validation<br/>[validate.py]"]
end

subgraph group_infra["Model and usage"]
  node_provider["Claude Code provider<br/>[claudecode.py]"]
  node_usage["Usage meter<br/>[usage.py]"]
end

node_operator(("Benchmark operator"))
node_claude{{"Claude Code CLI"}}

node_operator -->|"starts run"| node_hourly
node_operator -->|"starts run"| node_daily
node_hourly -->|"runs schedule"| node_scheduler
node_daily -->|"runs schedule"| node_scheduler
node_scheduler -->|"reads panel"| node_panel
node_scheduler -->|"dispatches tasks"| node_tasks
node_tasks -->|"builds items"| node_generators
node_generators -->|"dispatches family"| node_codegen
node_tasks -->|"scores responses"| node_scorers
node_tasks -->|"uses prompt helpers"| node_common
node_provider -->|"invokes CLI"| node_claude
node_tasks -.->|"requests model output"| node_provider
node_hourly -->|"checks budget"| node_usage
node_daily -->|"checks budget"| node_usage
node_scheduler -.->|"writes evaluations"| node_eval_logs
node_eval_logs -->|"reads samples"| node_analysis
node_analysis -->|"supplies statistics"| node_plot
node_calibration -->|"informs design"| node_design
node_design -->|"selects panel"| node_panel
node_operator -->|"runs validation"| node_validation
node_validation -->|"reports outcomes"| node_analysis
node_operator -->|"requests report"| node_prebaseline
node_operator -->|"requests plots"| node_preplot
node_prebaseline -->|"uses calibration"| node_calibration
node_preplot -->|"uses statistics"| node_analysis

click node_hourly "https://github.com/ninjahawk/livenerf/blob/main/livenerf/hourly.py"
click node_daily "https://github.com/ninjahawk/livenerf/blob/main/livenerf/daily.py"
click node_scheduler "https://github.com/ninjahawk/livenerf/blob/main/livenerf/schedule.py"
click node_panel "https://github.com/ninjahawk/livenerf/blob/main/data/standard_panel.json"
click node_tasks "https://github.com/ninjahawk/livenerf/blob/main/livenerf/tasks/suite.py"
click node_generators "https://github.com/ninjahawk/livenerf/tree/main/livenerf/generators"
click node_codegen "https://github.com/ninjahawk/livenerf/blob/main/livenerf/generators/code.py"
click node_scorers "https://github.com/ninjahawk/livenerf/blob/main/livenerf/scorers/checks.py"
click node_common "https://github.com/ninjahawk/livenerf/blob/main/livenerf/common.py"
click node_provider "https://github.com/ninjahawk/livenerf/blob/main/livenerf/providers/claudecode.py"
click node_usage "https://github.com/ninjahawk/livenerf/blob/main/livenerf/usage.py"
click node_analysis "https://github.com/ninjahawk/livenerf/blob/main/livenerf/analysis/__init__.py"
click node_plot "https://github.com/ninjahawk/livenerf/blob/main/livenerf/plot.py"
click node_design "https://github.com/ninjahawk/livenerf/blob/main/livenerf/design.py"
click node_calibration "https://github.com/ninjahawk/livenerf/tree/main/livenerf/benchmarks"
click node_validation "https://github.com/ninjahawk/livenerf/blob/main/livenerf/validate.py"
click node_prebaseline "https://github.com/ninjahawk/livenerf/blob/main/livenerf/report_prebaseline.py"
click node_preplot "https://github.com/ninjahawk/livenerf/blob/main/livenerf/plot_prebaseline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_hourly,node_daily,node_scheduler,node_panel toneBlue
class node_tasks,node_generators,node_codegen,node_scorers,node_common toneAmber
class node_eval_logs,node_analysis,node_plot,node_prebaseline,node_preplot toneMint
class node_design,node_calibration,node_validation toneRose
class node_provider,node_usage,node_operator,node_claude toneIndigo
```

## Components

### Run collection

- [Hourly runner](https://github.com/ninjahawk/livenerf/blob/main/livenerf/hourly.py): `livenerf/hourly.py`
- [Daily runner](https://github.com/ninjahawk/livenerf/blob/main/livenerf/daily.py): `livenerf/daily.py`
- [Stratified scheduler](https://github.com/ninjahawk/livenerf/blob/main/livenerf/schedule.py): `livenerf/schedule.py`
- [Frozen panel](https://github.com/ninjahawk/livenerf/blob/main/data/standard_panel.json): `data/standard_panel.json`

### Benchmark tasks

- [Inspect task suite](https://github.com/ninjahawk/livenerf/blob/main/livenerf/tasks/suite.py): `livenerf/tasks/suite.py`
- [Item generators](https://github.com/ninjahawk/livenerf/tree/main/livenerf/generators): `livenerf/generators`
- [Code tasks](https://github.com/ninjahawk/livenerf/blob/main/livenerf/generators/code.py): `livenerf/generators/code.py`
- [Task graders](https://github.com/ninjahawk/livenerf/blob/main/livenerf/scorers/checks.py): `livenerf/scorers/checks.py`
- [Shared prompts and seeds](https://github.com/ninjahawk/livenerf/blob/main/livenerf/common.py): `livenerf/common.py`

### Analysis and reporting

- **Inspect evaluation logs**
- [Window statistics](https://github.com/ninjahawk/livenerf/blob/main/livenerf/analysis/__init__.py): `livenerf/analysis/__init__.py`
- [Results charts](https://github.com/ninjahawk/livenerf/blob/main/livenerf/plot.py): `livenerf/plot.py`
- [Pre-baseline reporting](https://github.com/ninjahawk/livenerf/blob/main/livenerf/report_prebaseline.py): `livenerf/report_prebaseline.py`
- [Pre-baseline plots](https://github.com/ninjahawk/livenerf/blob/main/livenerf/plot_prebaseline.py): `livenerf/plot_prebaseline.py`

### Design and validation

- [Experimental design](https://github.com/ninjahawk/livenerf/blob/main/livenerf/design.py): `livenerf/design.py`
- [Benchmark calibration](https://github.com/ninjahawk/livenerf/tree/main/livenerf/benchmarks): `livenerf/benchmarks`
- [Instrument validation](https://github.com/ninjahawk/livenerf/blob/main/livenerf/validate.py): `livenerf/validate.py`

### Model and usage

- [Claude Code provider](https://github.com/ninjahawk/livenerf/blob/main/livenerf/providers/claudecode.py): `livenerf/providers/claudecode.py`
- [Usage meter](https://github.com/ninjahawk/livenerf/blob/main/livenerf/usage.py): `livenerf/usage.py`

### Other components

- **Benchmark operator**
- **Claude Code CLI**

## Connections

- Benchmark operator → Hourly runner: starts run (evidence: `README.md`)
- Benchmark operator → Daily runner: starts run (evidence: `README.md`)
- Hourly runner → Stratified scheduler: runs schedule (evidence: `livenerf/hourly.py`)
- Daily runner → Stratified scheduler: runs schedule (evidence: `livenerf/daily.py`)
- Stratified scheduler → Frozen panel: reads panel (evidence: `livenerf/schedule.py`)
- Stratified scheduler → Inspect task suite: dispatches tasks (evidence: `livenerf/schedule.py`)
- Inspect task suite → Item generators: builds items (evidence: `livenerf/tasks/suite.py`)
- Item generators → Code tasks: dispatches family (evidence: `livenerf/generators/__init__.py`)
- Inspect task suite → Task graders: scores responses (evidence: `livenerf/tasks/suite.py`)
- Inspect task suite → Shared prompts and seeds: uses prompt helpers (evidence: `livenerf/tasks/suite.py`)
- Claude Code provider → Claude Code CLI: invokes CLI (evidence: `README.md`)
- Inspect task suite → Claude Code provider: requests model output
- Hourly runner → Usage meter: checks budget (evidence: `livenerf/hourly.py`)
- Daily runner → Usage meter: checks budget (evidence: `livenerf/daily.py`)
- Stratified scheduler → Inspect evaluation logs: writes evaluations
- Inspect evaluation logs → Window statistics: reads samples (evidence: `livenerf/analysis/__init__.py`)
- Window statistics → Results charts: supplies statistics (evidence: `livenerf/plot.py`)
- Benchmark calibration → Experimental design: informs design (evidence: `livenerf/design.py`)
- Experimental design → Frozen panel: selects panel (evidence: `livenerf/design.py`)
- Benchmark operator → Instrument validation: runs validation (evidence: `livenerf/validate.py`)
- Instrument validation → Window statistics: reports outcomes (evidence: `livenerf/validate.py`)
- Benchmark operator → Pre-baseline reporting: requests report (evidence: `README.md`)
- Benchmark operator → Pre-baseline plots: requests plots (evidence: `README.md`)
- Pre-baseline reporting → Benchmark calibration: uses calibration (evidence: `livenerf/report_prebaseline.py`)
- Pre-baseline plots → Window statistics: uses statistics (evidence: `livenerf/plot_prebaseline.py`)
