How timme33/sentinel works
Sentinel turns Prometheus alerts into service-centered incidents, schedules investigations after a quiet period, gathers telemetry evidence, and produces validated, evidence-cited reports. The CLI assembles the runtime; detection and investigation can run as separate processes. The README documents the end-to-end flow and major capabilities. Source excerpts confirm alert conversion, polling, incident reconciliation, and selected CLI wiring; investigation internals and the OpenTelemetry Demo integration are represented from README statements and the repository tree rather than inspected source.
Main entry points: worker.py Investigation worker · cli.py Sentinel CLI · repository.py Investigation jobs · runner.py Detection runner
Components 21
Detection
- Alert source
src/detection/prometheus_alerts.py - Detection runner
src/detection/runner.py - RED metrics reader
src/telemetry/red_reader.py - RED query builder
src/telemetry/red_queries.py - Domain records
src/domain/models.py
Incidents and state
- Incident store
src/incidents/store.py - Investigation jobs
src/investigation/repository.py - SQLite state
Investigation
- Investigation worker
src/investigation/worker.py - Evidence collection
src/investigation/evidence_service.py - Context builder
src/investigation/context_builder.py - Codex agent
src/investigation/codex_agent.py - FastMCP tools
src/investigation/mcp_server.py - Report validator
src/investigation/report_validator.py
Telemetry
- Jaeger adapter
src/telemetry/jaeger.py - OpenSearch adapter
src/telemetry/opensearch.py - Prometheus adapter
src/telemetry/prometheus.py
Other components
- Operator
- Sentinel CLI
src/cli.py - Prometheus
- OpenTelemetry Demo
integration/otel-demo/compose.sentinel.yaml
Connections 33
- Operator → Sentinel CLI: operates
README.md - Sentinel CLI → Prometheus adapter: builds
src/cli.py - Sentinel CLI → Detection runner: starts detection
src/cli.py - Sentinel CLI → Investigation worker: starts worker
src/cli.py - Sentinel CLI → Incident store: reads and initializes
src/cli.py - Sentinel CLI → Investigation jobs: reads and initializes
src/cli.py - Detection runner → Alert source: polls
src/detection/runner.py - Alert source → Prometheus adapter: reads firing alerts
src/detection/prometheus_alerts.py - Prometheus adapter → Prometheus: queries
src/telemetry/prometheus.py - Alert source → Domain records: creates signals
src/detection/prometheus_alerts.py - Detection runner → Incident store: reconciles snapshot
src/detection/runner.py - Incident store → SQLite state: persists lifecycle
src/incidents/store.py - Detection runner → Investigation jobs: schedules jobs
src/detection/runner.py - Investigation jobs → SQLite state: persists jobs
README.md - Detection runner → Prometheus adapter: reports metrics
- Sentinel CLI → RED metrics reader: builds snapshot
src/cli.py - RED metrics reader → RED query builder: uses queries
src/telemetry/red_reader.py - RED metrics reader → Prometheus adapter: queries metrics
src/telemetry/red_reader.py - OpenTelemetry Demo → Prometheus: exports telemetry
README.md - Prometheus → Alert source: provides alerts
README.md - Investigation worker → Investigation jobs: claims and updates
README.md - Investigation worker → Incident store: loads incident
README.md - Investigation worker → Evidence collection: collects evidence
README.md - Evidence collection → Jaeger adapter: queries traces
README.md - Evidence collection → OpenSearch adapter: queries logs
README.md - Evidence collection → Investigation jobs: stores evidence
README.md - Investigation worker → Context builder: builds context
README.md - Investigation worker → Codex agent: invokes agent
README.md - Codex agent → FastMCP tools: requests evidence
README.md - FastMCP tools → Jaeger adapter: searches traces
README.md - FastMCP tools → OpenSearch adapter: searches logs
README.md - Investigation worker → Report validator: validates report
README.md - Report validator → Investigation jobs: persists report
README.md