How dch2r/signalfloor works
signalfloor helps teams distinguish real evaluation regressions from noise, assess judge reliability, and measure run-to-run flakiness. The CLI compares case-paired scores, calibrates judge labels, and analyzes repeated runs; the server and dashboard add stored runs, comparisons, jobs, and OpenTelemetry traces. A reference agent supports retrieval-backed evaluation. The map emphasizes documented capabilities, but some server, dashboard, and agent connections are inferred only at the product level because their wiring was not sampled.
Main entry points: cli.py CLI commands · compare.py Run comparison · app.py HTTP application · calibration.py Judge calibration
Components 23
Statistical analysis
- CLI commands
packages/signalfloor/src/signalfloor/cli.py - Run comparison
packages/signalfloor/src/signalfloor/compare.py - Betting bounds
packages/signalfloor/src/signalfloor/_betting.py - Case pairing
packages/signalfloor/src/signalfloor/_pairing.py - Resampling
packages/signalfloor/src/signalfloor/_resample.py - Judge calibration
packages/signalfloor/src/signalfloor/calibration.py - Flakiness analysis
packages/signalfloor/src/signalfloor/stability.py - Result formatting
packages/signalfloor/src/signalfloor/results.py
Reference agent
- Reference agent
packages/agent/src/signalfloor_agent/agent.py - Passage retrieval
packages/agent/src/signalfloor_agent/retrieval.py - Embedding client
packages/agent/src/signalfloor_agent/embeddings.py
Run data service
- HTTP application
packages/server/src/signalfloor_server/app.py - Run endpoints
packages/server/src/signalfloor_server/runs.py - Stored comparisons
packages/server/src/signalfloor_server/comparisons.py - Job endpoints
packages/server/src/signalfloor_server/job_routes.py - Job processing
packages/server/src/signalfloor_server/jobs.py - Trace ingestion
packages/server/src/signalfloor_server/traces.py - Run and span records
packages/server/src/signalfloor_server/models.py
Dashboard
- Web dashboard
dashboard/src/app/page.tsx - Server API client
dashboard/src/lib/api.ts - API types
dashboard/src/lib/api-types.ts
Other components
- Eval practitioner
- OTel exporters
Connections 27
- Eval practitioner → CLI commands: runs checks
README.md - CLI commands → Run comparison: dispatches
packages/signalfloor/src/signalfloor/cli.py - CLI commands → Judge calibration: dispatches
packages/signalfloor/src/signalfloor/cli.py - CLI commands → Flakiness analysis: dispatches
packages/signalfloor/src/signalfloor/cli.py - CLI commands → Result formatting: formats results
packages/signalfloor/src/signalfloor/cli.py - Run comparison → Case pairing: pairs cases
packages/signalfloor/src/signalfloor/compare.py - Run comparison → Betting bounds: computes bounds
packages/signalfloor/src/signalfloor/compare.py - Run comparison → Resampling: bootstraps interval
packages/signalfloor/src/signalfloor/compare.py - Judge calibration → Case pairing: aligns labels
packages/signalfloor/src/signalfloor/calibration.py - Judge calibration → Resampling: resamples agreement
packages/signalfloor/src/signalfloor/calibration.py - Flakiness analysis → Case pairing: aligns cases
packages/signalfloor/src/signalfloor/stability.py - Eval practitioner → Reference agent: evaluates questions
README.md - Reference agent → Passage retrieval: retrieves passages
packages/agent/src/signalfloor_agent/agent.py - Passage retrieval → Embedding client: embeds content
packages/agent/src/signalfloor_agent/retrieval.py - Eval practitioner → Web dashboard: reviews runs
README.md - Web dashboard → Server API client: loads views
- Server API client → API types: uses schemas
dashboard/src/lib/api.ts - Server API client → HTTP application: requests API
dashboard/src/lib/api.ts - HTTP application → Run endpoints: registers routes
packages/server/src/signalfloor_server/app.py - HTTP application → Trace ingestion: registers routes
packages/server/src/signalfloor_server/app.py - HTTP application → Job endpoints: registers routes
packages/server/src/signalfloor_server/app.py - Run endpoints → Run and span records: queries records
packages/server/src/signalfloor_server/runs.py - Run endpoints → Stored comparisons: computes comparisons
packages/server/src/signalfloor_server/runs.py - Job endpoints → Job processing: enqueues work
packages/server/src/signalfloor_server/job_routes.py - Job processing → Stored comparisons: runs comparisons
packages/server/src/signalfloor_server/jobs.py - Trace ingestion → Run and span records: stores spans
packages/server/src/signalfloor_server/traces.py - OTel exporters → Trace ingestion: exports traces
packages/server/src/signalfloor_server/traces.py