How tushaarrr/regresgate works
regressgate is a statistical regression gate around promptfoo evals. CI or an engineer runs the fixed eval suite, promptfoo invokes the support provider and produces exports, then baseline selection and pairing establish comparable per-case observations. The gate checks infrastructure validity, exact directional McNemar significance, confidence bounds, exclusions, and power guardrails before returning pass/comment/block/refuse. Supporting branches monitor drift, triage failures, pool deliberate adjudication reruns, cache verdicts, and compare production escapes. Several implementation files were sampled; unsampled connections are shown only where the README or CLI workflow documents them.
Components 21
Evaluation Runtime
- Eval Suite
eval - Support Provider
eval/provider.py - Offline Provider
eval/offline/provider.py
Comparison Pipeline
- Eval Export
- Baseline Fetcher
regressgate/fetch_baseline.py - Case Pairing
regressgate/pair.py - Run Store
regressgate/store.py - Gate Decision
regressgate/gate.py - Stats Adapter
regressgate/stats/adapter.py - Paired Statistics
regressgate/stats/paired.py
Guardrails
- Verdict Cache
regressgate/verdict_cache.py - Quarantine Analysis
regressgate/quarantine.py - Drift Monitor
regressgate/drift_monitor.py
Operational Review
- Failure Triage
regressgate/triage.py - Pooled Adjudication
regressgate/adjudicate.py - Escape Monitor
regressgate/escape_monitor.py - Incident Log
Other components
- CI Engineer
- Promptfoo Runner
- Model API
- Merge Reviewer
Connections 29
- CI Engineer → Promptfoo Runner: runs eval
- Promptfoo Runner → Eval Suite: loads config
- Promptfoo Runner → Support Provider: invokes provider
- Promptfoo Runner → Offline Provider: invokes provider
- Support Provider → Model API: calls API
- Promptfoo Runner → Eval Export: writes export
- Eval Export → Run Store: ingests results
- CI Engineer → Baseline Fetcher: finds baseline
- Baseline Fetcher → Eval Export: fetches baseline
- Eval Export → Case Pairing: supplies exports
- Run Store → Case Pairing: supplies run context
- Quarantine Analysis → Case Pairing: supplies exclusions
- Case Pairing → Gate Decision: writes pairing
- Gate Decision → Stats Adapter: computes test
- Stats Adapter → Paired Statistics: calls statistics
- Gate Decision → Verdict Cache: reads or writes
- Quarantine Analysis → Run Store: reads replays
- Drift Monitor → Run Store: reads runs
- Failure Triage → Run Store: reads failures
- Failure Triage → Case Pairing: reads pairing
- Pooled Adjudication → Case Pairing: pools replicates
- Pooled Adjudication → Incident Log: logs outcome
- Merge Reviewer → Pooled Adjudication: requests review
- Merge Reviewer → Escape Monitor: records incident
- Escape Monitor → Incident Log: appends incident
- Escape Monitor → Verdict Cache: joins verdicts
- Gate Decision → Merge Reviewer: reports verdict
- Failure Triage → Merge Reviewer: adds explanations
- Drift Monitor → Merge Reviewer: raises alarms