# balasnest/agent_reliability_lab architecture

> Architecture diagram of the GitHub repository balasnest/agent_reliability_lab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/balasnest/agent_reliability_lab
- Repository: https://github.com/balasnest/agent_reliability_lab
- Video tour (about a minute): https://gitdiagram.com/balasnest/agent_reliability_lab/video

## Overview

Agent Reliability Lab is a reviewer console and offline reliability harness for a ShopNova support agent. In the live workflow, a reviewer selects a customer or scenario and sends a message; the console streams the API response, trace events, and per-turn evaluation. The API invokes the shared LangGraph agent, whose tools handle customer/order lookups, knowledge retrieval, and escalation. The evaluation runner replays scripted scenarios through that graph and scores outcomes with five evaluators. The README documents these branches; sampled source confirms key API, graph, retrieval, and persistence behavior, while some tool and console wiring is shown only at subsystem level.

Main entry points: [`backend/agent/graph.py`](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/graph.py) (LangGraph agent), [`backend/db.py`](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/db.py) (Postgres and pgvector), [`backend/api/main.py`](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/api/main.py) (FastAPI service), [`backend/evaluation/runner.py`](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/evaluation/runner.py) (Evaluation runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_console["Reviewer console"]
  node_console_app["Console UI"]
  node_chat_ui["Chat and trace<br/>[ChatPanel.tsx]"]
  node_context_ui["Customer and docs<br/>[TopBar.tsx]"]
  node_scenario_ui["Scenario library<br/>[ScenarioRail.tsx]"]
  node_frontend_api["API client<br/>[api.ts]"]
end

subgraph group_service["API and agent"]
  node_api["FastAPI service<br/>[main.py]"]
  node_agent["LangGraph agent<br/>[graph.py]"]
end

subgraph group_capabilities["Support capabilities"]
  node_order_tools["Order tools<br/>[order_tools.py]"]
  node_customer_tools["Customer tools<br/>[customer_tools.py]"]
  node_kb_tools["Knowledge tool<br/>[kb_tools.py]"]
  node_escalation_tools["Escalation tools"]
end

subgraph group_knowledge["Knowledge and data"]
  node_ingest["Document ingestion<br/>[ingest.py]"]
  node_retriever["Vector retriever<br/>[retriever.py]"]
  node_documents["KB documents"]
  node_database[("Postgres and pgvector<br/>[db.py]")]
end

subgraph group_reliability["Reliability evaluation"]
  node_eval_runner["Evaluation runner<br/>[runner.py]"]
  node_scenarios["Scenario corpus<br/>[scenarios.json]"]
  node_evaluators["Five evaluators"]
  node_eval_report["Evaluation report"]
end

node_reviewer(("Reviewer"))
node_openai(("OpenAI model"))

node_reviewer -->|"uses"| node_console_app
node_console_app -->|"renders"| node_chat_ui
node_console_app -->|"renders"| node_context_ui
node_console_app -->|"renders"| node_scenario_ui
node_chat_ui -->|"streams chat"| node_frontend_api
node_context_ui -->|"loads context"| node_frontend_api
node_scenario_ui -->|"loads scenarios"| node_frontend_api
node_frontend_api -->|"HTTP requests"| node_api
node_api -->|"streams graph"| node_agent
node_agent -.->|"calls model"| node_openai
node_agent -->|"dispatches"| node_order_tools
node_agent -->|"dispatches"| node_customer_tools
node_agent -->|"dispatches"| node_kb_tools
node_agent -->|"dispatches"| node_escalation_tools
node_order_tools -->|"reads orders"| node_database
node_customer_tools -->|"reads profiles"| node_database
node_escalation_tools -->|"writes tickets"| node_database
node_kb_tools -->|"searches"| node_retriever
node_retriever -->|"queries vectors"| node_database
node_documents -->|"supplies text"| node_ingest
node_ingest -->|"stores embeddings"| node_database
node_api -->|"serves REST data"| node_database
node_api -->|"scores each turn"| node_evaluators
node_eval_runner -->|"reads cases"| node_scenarios
node_eval_runner -->|"replays turns"| node_agent
node_eval_runner -->|"scores runs"| node_evaluators
node_eval_runner -->|"writes results"| node_eval_report
node_scenario_ui -->|"requests cases"| node_api

click node_console_app "https://github.com/balasnest/agent_reliability_lab/tree/main/frontend/app/console"
click node_chat_ui "https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/components/ChatPanel.tsx"
click node_context_ui "https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/components/TopBar.tsx"
click node_scenario_ui "https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/components/ScenarioRail.tsx"
click node_frontend_api "https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/lib/api.ts"
click node_api "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/api/main.py"
click node_agent "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/graph.py"
click node_order_tools "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/order_tools.py"
click node_customer_tools "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/customer_tools.py"
click node_kb_tools "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/kb_tools.py"
click node_escalation_tools "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/escalation_tools.py"
click node_ingest "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/knowledge_base/ingest.py"
click node_retriever "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/knowledge_base/retriever.py"
click node_documents "https://github.com/balasnest/agent_reliability_lab/tree/main/backend/knowledge_base/documents"
click node_database "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/db.py"
click node_eval_runner "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/evaluation/runner.py"
click node_scenarios "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/data/scenarios/scenarios.json"
click node_evaluators "https://github.com/balasnest/agent_reliability_lab/tree/main/backend/evaluation/evaluators"
click node_eval_report "https://github.com/balasnest/agent_reliability_lab/blob/main/backend/evaluation/evaluation_report.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_console_app,node_chat_ui,node_context_ui,node_scenario_ui,node_frontend_api,node_reviewer toneBlue
class node_api,node_agent toneAmber
class node_order_tools,node_customer_tools,node_kb_tools,node_escalation_tools toneMint
class node_ingest,node_retriever,node_documents,node_database toneRose
class node_eval_runner,node_scenarios,node_evaluators,node_eval_report,node_openai toneIndigo
```

## Components

### Reviewer console

- [Console UI](https://github.com/balasnest/agent_reliability_lab/tree/main/frontend/app/console): `frontend/app/console`
- [Chat and trace](https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/components/ChatPanel.tsx): `frontend/components/ChatPanel.tsx`
- [Customer and docs](https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/components/TopBar.tsx): `frontend/components/TopBar.tsx`
- [Scenario library](https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/components/ScenarioRail.tsx): `frontend/components/ScenarioRail.tsx`
- [API client](https://github.com/balasnest/agent_reliability_lab/blob/main/frontend/lib/api.ts): `frontend/lib/api.ts`

### API and agent

- [FastAPI service](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/api/main.py): `backend/api/main.py`
- [LangGraph agent](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/graph.py): `backend/agent/graph.py`

### Support capabilities

- [Order tools](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/order_tools.py): `backend/agent/tools/order_tools.py`
- [Customer tools](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/customer_tools.py): `backend/agent/tools/customer_tools.py`
- [Knowledge tool](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/kb_tools.py): `backend/agent/tools/kb_tools.py`
- [Escalation tools](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/agent/tools/escalation_tools.py): `backend/agent/tools/escalation_tools.py`

### Knowledge and data

- [Document ingestion](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/knowledge_base/ingest.py): `backend/knowledge_base/ingest.py`
- [Vector retriever](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/knowledge_base/retriever.py): `backend/knowledge_base/retriever.py`
- [KB documents](https://github.com/balasnest/agent_reliability_lab/tree/main/backend/knowledge_base/documents): `backend/knowledge_base/documents`
- [Postgres and pgvector](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/db.py): `backend/db.py`

### Reliability evaluation

- [Evaluation runner](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/evaluation/runner.py): `backend/evaluation/runner.py`
- [Scenario corpus](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/data/scenarios/scenarios.json): `backend/data/scenarios/scenarios.json`
- [Five evaluators](https://github.com/balasnest/agent_reliability_lab/tree/main/backend/evaluation/evaluators): `backend/evaluation/evaluators`
- [Evaluation report](https://github.com/balasnest/agent_reliability_lab/blob/main/backend/evaluation/evaluation_report.json): `backend/evaluation/evaluation_report.json`

### Other components

- **Reviewer**
- **OpenAI model**

## Connections

- Reviewer → Console UI: uses
- Console UI → Chat and trace: renders
- Console UI → Customer and docs: renders
- Console UI → Scenario library: renders
- Chat and trace → API client: streams chat
- Customer and docs → API client: loads context
- Scenario library → API client: loads scenarios
- API client → FastAPI service: HTTP requests
- FastAPI service → LangGraph agent: streams graph
- LangGraph agent → OpenAI model: calls model
- LangGraph agent → Order tools: dispatches
- LangGraph agent → Customer tools: dispatches
- LangGraph agent → Knowledge tool: dispatches
- LangGraph agent → Escalation tools: dispatches
- Order tools → Postgres and pgvector: reads orders
- Customer tools → Postgres and pgvector: reads profiles
- Escalation tools → Postgres and pgvector: writes tickets
- Knowledge tool → Vector retriever: searches
- Vector retriever → Postgres and pgvector: queries vectors
- KB documents → Document ingestion: supplies text
- Document ingestion → Postgres and pgvector: stores embeddings
- FastAPI service → Postgres and pgvector: serves REST data
- FastAPI service → Five evaluators: scores each turn
- Evaluation runner → Scenario corpus: reads cases
- Evaluation runner → LangGraph agent: replays turns
- Evaluation runner → Five evaluators: scores runs
- Evaluation runner → Evaluation report: writes results
- Scenario library → FastAPI service: requests cases