sarancoding/production-reliability-deployment-field-engineering-suite

Purpose and Entry Points

Field Engineering Suite is a Python 3.11+ operational platform for isolated customer deployment, SLA monitoring, incident automation, and publication of postmortems/case studies. Primary operator entry points are Makefile (make docker-up, make deploy) and the API route layer under src/api/routes/. Runtime configuration begins in .env.example and src/core/config.py.

Lifecycle Architecture

The intended control flow is: provision → monitor → detect → respond → learn → share. Deployment hands environments to health monitoring; SLA breaches become events; incidents consume those events; resolution produces postmortems and material for externally reviewed case studies. The source package is organized around these bounded lifecycle domains:

• Deployment: src/deploy/__init__.py

• Health/SLA: src/health/__init__.py

• Incident response: src/incident/__init__.py

• Case-study publishing: src/casestudy/__init__.py

API and Core Runtime Boundary

FastAPI/Pydantic/Uvicorn expose operational functions through src/api/routes/__init__.py. Cross-cutting runtime concerns are centralized rather than embedded in lifecycle domains: configuration in src/core/config.py, standardized failures in src/core/exceptions.py, structured logs in src/core/logging.py, and distributed trace propagation in src/core/tracing.py. Trace IDs are a key boundary-crossing correlation mechanism across deployment, monitoring, and incident workflows.

Deployment and Isolation

The deployment domain (src/deploy/__init__.py) validates customer/environment inputs, applies infrastructure, gates traffic on health, registers successful customers, and rolls back failed deployments. The README specifies Terraform plus Kubernetes, including per-customer namespaces and network policies; remote Terraform state locking is expected infrastructure behavior. Production application requires a human approval gate, unlike lower environments.

Health, SLA, and Event Transport

Health processing in src/health/__init__.py aggregates customer metrics, calculates error-budget burn rates, and identifies SLA violations. Fast burn alerts page immediately; slow burn creates planning work. Domain-to-domain communication is mediated by the event bus at src/events/bus.py, intended to provide transactional-outbox-style reliable delivery rather than direct health-to-incident coupling. Event contracts belong under src/events/__init__.py.

Incident Learning and Publication

Incident processing (src/incident/__init__.py) classifies SEV1–SEV4, opens and manages mitigation/escalation loops, uses traces and metrics for diagnosis, and generates a blameless postmortem after resolution. Generated artifacts are retained in postmortems/generated/; durable improvement work and lessons flow into tasks/todo.md and tasks/lessons.md. Case-study assembly (src/casestudy/__init__.py) gathers approved operational material and requires human review before publication. PDF generation is supported by scripts/generate_pdfs.py into docs/pdfs/.

Material Services and Operations

The full local stack comprises the app, PostgreSQL, Redis, OpenTelemetry/OTLP, Prometheus, and Grafana. PostgreSQL is the system of record; Redis supports caching or coordination; OpenTelemetry links signals; Prometheus/Grafana provide metrics and dashboarding. Reliability validation spans unit, integration, and failure-injection tiers under tests/unit/, tests/integration/, and tests/chaos/, with chaos gates specifically validating rollback, breach-to-incident, postmortem generation, and end-to-end trace propagation.