# eternal-drift/production-incident-copilot architecture

> Architecture diagram of the GitHub repository eternal-drift/production-incident-copilot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eternal-drift/production-incident-copilot
- Repository: https://github.com/eternal-drift/production-incident-copilot

## Overview

Incident Copilot is an authenticated on-call assistant: operators submit questions or incident records, and the app returns grounded runbook answers, checks live incident state, or escalates when evidence is insufficient. Its main answer path runs through the chat router and LangGraph workflow; runbook retrieval uses hybrid Postgres search and reranking. The product also supports incident CRUD and human-reviewed post-incident drafts. Some workflow and live-lookup implementation was not sampled, so those documented links are shown with README evidence; optional integrations are separated.

Main entry points: [`app/ai/rag.py`](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/rag.py) (Hybrid retrieval), [`app/routers/incidents.py`](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/routers/incidents.py) (Incident API), [`app/routers/chat.py`](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/routers/chat.py) (Chat endpoint), [`app/graph/workflow.py`](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/graph/workflow.py) (Decision workflow)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and access"]
  node_chat_api["Chat endpoint<br/>[chat.py]"]
  node_incident_api["Incident API<br/>[incidents.py]"]
  node_auth["API-key auth<br/>[auth.py]"]
end

subgraph group_answer["Answer workflow"]
  node_workflow["Decision workflow<br/>[workflow.py]"]
  node_live_lookup["Live incident lookup<br/>[live_tool.py]"]
end

subgraph group_knowledge["Runbook retrieval"]
  node_rag["Hybrid retrieval<br/>[rag.py]"]
  node_embeddings["Embedding client<br/>[embeddings.py]"]
  node_reranker["Cross-encoder reranker<br/>[reranker.py]"]
  node_runbooks["Runbook corpus"]
end

subgraph group_incidents["Incident operations"]
  node_post_review["Review draft generator<br/>[post_incident.py]"]
end

subgraph group_platform["State and integrations"]
  node_models["Incident and chunk models<br/>[models.py]"]
  node_database["Database access<br/>[database.py]"]
  node_postgres[("Postgres + pgvector")]
  node_llm["LLM client<br/>[llm_client.py]"]
  node_mcp_client["MCP client<br/>[mcp_client.py]"]
  node_mcp_server["MCP server<br/>[mcp_server.py]"]
end

node_operator(("On-call operator"))
node_ollama["Ollama"]
node_litellm["LiteLLM gateway"]

node_operator -->|"asks question"| node_chat_api
node_operator -->|"manages incidents"| node_incident_api
node_chat_api -->|"requires key"| node_auth
node_incident_api -->|"requires key"| node_auth
node_chat_api -->|"invokes"| node_workflow
node_workflow -->|"retrieves evidence"| node_rag
node_workflow -->|"checks live state"| node_live_lookup
node_rag -->|"embeds query"| node_embeddings
node_rag -->|"reranks candidates"| node_reranker
node_rag -->|"indexes documents"| node_runbooks
node_rag -->|"searches chunks"| node_postgres
node_incident_api -->|"requests draft"| node_post_review
node_post_review -->|"retrieves context"| node_rag
node_post_review -->|"generates draft"| node_llm
node_incident_api -->|"reads and writes"| node_models
node_database -->|"connects to"| node_postgres
node_models -->|"declares tables"| node_database
node_embeddings -->|"requests embeddings"| node_ollama
node_llm -->|"generates text"| node_ollama
node_llm -.->|"routes optionally"| node_litellm
node_mcp_client -.->|"calls optionally"| node_mcp_server
node_mcp_server -->|"wraps lookup"| node_live_lookup

click node_chat_api "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/routers/chat.py"
click node_incident_api "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/routers/incidents.py"
click node_auth "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/auth.py"
click node_workflow "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/graph/workflow.py"
click node_live_lookup "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/live_tool.py"
click node_rag "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/rag.py"
click node_embeddings "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/embeddings.py"
click node_reranker "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/reranker.py"
click node_runbooks "https://github.com/eternal-drift/production-incident-copilot/tree/main/sample_data/runbooks"
click node_post_review "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/post_incident.py"
click node_models "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/models.py"
click node_database "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/database.py"
click node_llm "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/llm_client.py"
click node_mcp_client "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/mcp_client.py"
click node_mcp_server "https://github.com/eternal-drift/production-incident-copilot/blob/main/app/mcp_server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chat_api,node_incident_api,node_auth toneBlue
class node_workflow,node_live_lookup toneAmber
class node_rag,node_embeddings,node_reranker,node_runbooks toneMint
class node_post_review toneRose
class node_models,node_database,node_postgres,node_llm,node_mcp_client,node_mcp_server,node_operator,node_ollama,node_litellm toneIndigo
```

## Components

### API and access

- [Chat endpoint](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/routers/chat.py): `app/routers/chat.py`
- [Incident API](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/routers/incidents.py): `app/routers/incidents.py`
- [API-key auth](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/auth.py): `app/auth.py`

### Answer workflow

- [Decision workflow](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/graph/workflow.py): `app/graph/workflow.py`
- [Live incident lookup](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/live_tool.py): `app/ai/live_tool.py`

### Runbook retrieval

- [Hybrid retrieval](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/rag.py): `app/ai/rag.py`
- [Embedding client](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/embeddings.py): `app/ai/embeddings.py`
- [Cross-encoder reranker](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/reranker.py): `app/ai/reranker.py`
- [Runbook corpus](https://github.com/eternal-drift/production-incident-copilot/tree/main/sample_data/runbooks): `sample_data/runbooks`

### Incident operations

- [Review draft generator](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/post_incident.py): `app/ai/post_incident.py`

### State and integrations

- [Incident and chunk models](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/models.py): `app/models.py`
- [Database access](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/database.py): `app/database.py`
- **Postgres + pgvector**
- [LLM client](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/llm_client.py): `app/ai/llm_client.py`
- [MCP client](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/ai/mcp_client.py): `app/ai/mcp_client.py`
- [MCP server](https://github.com/eternal-drift/production-incident-copilot/blob/main/app/mcp_server.py): `app/mcp_server.py`

### Other components

- **On-call operator**
- **Ollama**
- **LiteLLM gateway**

## Connections

- On-call operator → Chat endpoint: asks question (evidence: `README.md`)
- On-call operator → Incident API: manages incidents (evidence: `README.md`)
- Chat endpoint → API-key auth: requires key (evidence: `app/routers/chat.py`)
- Incident API → API-key auth: requires key (evidence: `app/routers/incidents.py`)
- Chat endpoint → Decision workflow: invokes (evidence: `app/routers/chat.py`)
- Decision workflow → Hybrid retrieval: retrieves evidence (evidence: `README.md`)
- Decision workflow → Live incident lookup: checks live state (evidence: `README.md`)
- Hybrid retrieval → Embedding client: embeds query (evidence: `app/ai/rag.py`)
- Hybrid retrieval → Cross-encoder reranker: reranks candidates (evidence: `app/ai/rag.py`)
- Hybrid retrieval → Runbook corpus: indexes documents (evidence: `app/ai/rag.py`)
- Hybrid retrieval → Postgres + pgvector: searches chunks (evidence: `app/ai/rag.py`)
- Incident API → Review draft generator: requests draft (evidence: `app/routers/incidents.py`)
- Review draft generator → Hybrid retrieval: retrieves context (evidence: `app/ai/post_incident.py`)
- Review draft generator → LLM client: generates draft (evidence: `app/ai/post_incident.py`)
- Incident API → Incident and chunk models: reads and writes (evidence: `app/routers/incidents.py`)
- Database access → Postgres + pgvector: connects to (evidence: `app/database.py`)
- Incident and chunk models → Database access: declares tables (evidence: `app/models.py`)
- Embedding client → Ollama: requests embeddings (evidence: `app/ai/embeddings.py`)
- LLM client → Ollama: generates text (evidence: `app/ai/llm_client.py`)
- LLM client → LiteLLM gateway: routes optionally (evidence: `app/ai/llm_client.py`)
- MCP client → MCP server: calls optionally (evidence: `README.md`)
- MCP server → Live incident lookup: wraps lookup (evidence: `app/mcp_server.py`)
