How balasnest/agent_reliability_lab works
Agent Reliability Lab is a reviewer console and offline reliability harness for a ShopNova support agent. In the live workflow, a reviewer selects a customer or scenario and sends a message; the console streams the API response, trace events, and per-turn evaluation. The API invokes the shared LangGraph agent, whose tools handle customer/order lookups, knowledge retrieval, and escalation. The evaluation runner replays scripted scenarios through that graph and scores outcomes with five evaluators. The README documents these branches; sampled source confirms key API, graph, retrieval, and persistence behavior, while some tool and console wiring is shown only at subsystem level.
Main entry points: graph.py LangGraph agent · db.py Postgres and pgvector · main.py FastAPI service · runner.py Evaluation runner
Components 21
Reviewer console
- Console UI
frontend/app/console - Chat and trace
frontend/components/ChatPanel.tsx - Customer and docs
frontend/components/TopBar.tsx - Scenario library
frontend/components/ScenarioRail.tsx - API client
frontend/lib/api.ts
API and agent
- FastAPI service
backend/api/main.py - LangGraph agent
backend/agent/graph.py
Support capabilities
- Order tools
backend/agent/tools/order_tools.py - Customer tools
backend/agent/tools/customer_tools.py - Knowledge tool
backend/agent/tools/kb_tools.py - Escalation tools
backend/agent/tools/escalation_tools.py
Knowledge and data
- Document ingestion
backend/knowledge_base/ingest.py - Vector retriever
backend/knowledge_base/retriever.py - KB documents
backend/knowledge_base/documents - Postgres and pgvector
backend/db.py
Reliability evaluation
- Evaluation runner
backend/evaluation/runner.py - Scenario corpus
backend/data/scenarios/scenarios.json - Five evaluators
backend/evaluation/evaluators - Evaluation report
backend/evaluation/evaluation_report.json
Other components
- Reviewer
- OpenAI model
Connections 28
- Reviewer → Console UI: uses
- Console UI → Chat and trace: renders
- Console UI → Customer and docs: renders
- Console UI → Scenario library: renders
- Chat and trace → API client: streams chat
- Customer and docs → API client: loads context
- Scenario library → API client: loads scenarios
- API client → FastAPI service: HTTP requests
- FastAPI service → LangGraph agent: streams graph
- LangGraph agent → OpenAI model: calls model
- LangGraph agent → Order tools: dispatches
- LangGraph agent → Customer tools: dispatches
- LangGraph agent → Knowledge tool: dispatches
- LangGraph agent → Escalation tools: dispatches
- Order tools → Postgres and pgvector: reads orders
- Customer tools → Postgres and pgvector: reads profiles
- Escalation tools → Postgres and pgvector: writes tickets
- Knowledge tool → Vector retriever: searches
- Vector retriever → Postgres and pgvector: queries vectors
- KB documents → Document ingestion: supplies text
- Document ingestion → Postgres and pgvector: stores embeddings
- FastAPI service → Postgres and pgvector: serves REST data
- FastAPI service → Five evaluators: scores each turn
- Evaluation runner → Scenario corpus: reads cases
- Evaluation runner → LangGraph agent: replays turns
- Evaluation runner → Five evaluators: scores runs
- Evaluation runner → Evaluation report: writes results
- Scenario library → FastAPI service: requests cases