# shreyasgaikwad-04/llm-evals-and-guardrails architecture

> Architecture diagram of the GitHub repository shreyasgaikwad-04/llm-evals-and-guardrails, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shreyasgaikwad-04/llm-evals-and-guardrails
- Repository: https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails

## Overview

The platform takes a dataset and model configuration through a Streamlit dashboard or API, validates records, generates model answers, evaluates quality and guardrails, and stores evaluation runs for review and regression comparison. The graph separates the user interface, API orchestration, evaluation capabilities, and persistence. Optional provider-backed evaluation is distinguished from deterministic fallback behavior. The README documents dashboard and metric capabilities; unsampled dashboard internals and database models limit detail about their implementation.

Main entry points: [`app/api/routes.py`](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/api/routes.py) (API routes), [`app/services/evaluation_service.py`](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/evaluation_service.py) (Evaluation runner), [`app/services/llm_service.py`](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/llm_service.py) (LLM generation), [`dashboard/streamlit_app.py`](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/dashboard/streamlit_app.py) (Streamlit dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_dashboard["Streamlit dashboard<br/>[streamlit_app.py]"]
  node_client["HTTP client<br/>[api_client.py]"]
end

subgraph group_api["API orchestration"]
  node_api_routes["API routes<br/>[routes.py]"]
  node_main["FastAPI application<br/>[main.py]"]
end

subgraph group_evaluation["Evaluation capabilities"]
  node_dataset["Dataset loader<br/>[dataset_loader.py]"]
  node_evaluation_service["Evaluation runner"]
  node_llm["LLM generation<br/>[llm_service.py]"]
  node_cost["Cost calculator<br/>[cost_calculator.py]"]
  node_quality["Quality metrics<br/>[quality_service.py]"]
  node_judge["Structured judge<br/>[judge_service.py]"]
  node_judge_prompts["Judge prompts"]
  node_guardrails["Guardrail checks"]
  node_regression["Regression comparison"]
end

subgraph group_persistence["Persistence"]
  node_database[("Evaluation store<br/>[database.py]")]
end

node_user(("Engineer"))
node_openai{{"OpenAI"}}
node_deepeval{{"DeepEval"}}

node_user -->|"uses"| node_dashboard
node_dashboard -->|"requests"| node_client
node_client -->|"HTTP calls"| node_api_routes
node_main -->|"mounts"| node_api_routes
node_api_routes -->|"loads records"| node_dataset
node_api_routes -->|"runs evaluation"| node_evaluation_service
node_evaluation_service -->|"screens input"| node_guardrails
node_evaluation_service -->|"generates answers"| node_llm
node_llm -.->|"requests completion"| node_openai
node_llm -->|"estimates cost"| node_cost
node_evaluation_service -->|"scores answers"| node_quality
node_quality -.->|"computes metrics"| node_deepeval
node_quality -->|"requests judge score"| node_judge
node_judge -.->|"requests structured score"| node_openai
node_judge -->|"uses templates"| node_judge_prompts
node_api_routes -->|"runs suite"| node_guardrails
node_guardrails -.->|"tests resistance"| node_llm
node_api_routes -->|"compares runs"| node_regression
node_api_routes -->|"reads and writes runs"| node_database
node_dashboard -->|"views run summaries"| node_client

click node_dashboard "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/dashboard/streamlit_app.py"
click node_client "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/dashboard/api_client.py"
click node_api_routes "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/api/routes.py"
click node_dataset "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/utils/dataset_loader.py"
click node_evaluation_service "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/evaluation_service.py"
click node_llm "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/llm_service.py"
click node_cost "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/utils/cost_calculator.py"
click node_quality "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/quality_service.py"
click node_judge "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/judge_service.py"
click node_judge_prompts "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/prompts/evaluation_prompts.py"
click node_guardrails "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/guardrail_service.py"
click node_regression "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/regression_service.py"
click node_database "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/db/database.py"
click node_main "https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_client toneBlue
class node_api_routes,node_main toneAmber
class node_dataset,node_evaluation_service,node_llm,node_cost,node_quality,node_judge,node_judge_prompts,node_guardrails,node_regression toneMint
class node_database toneRose
class node_user,node_openai,node_deepeval toneIndigo
```

## Components

### User interface

- [Streamlit dashboard](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/dashboard/streamlit_app.py): `dashboard/streamlit_app.py`
- [HTTP client](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/dashboard/api_client.py): `dashboard/api_client.py`

### API orchestration

- [API routes](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/api/routes.py): `app/api/routes.py`
- [FastAPI application](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/main.py): `app/main.py`

### Evaluation capabilities

- [Dataset loader](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/utils/dataset_loader.py): `app/utils/dataset_loader.py`
- [Evaluation runner](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/evaluation_service.py): `app/services/evaluation_service.py`
- [LLM generation](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/llm_service.py): `app/services/llm_service.py`
- [Cost calculator](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/utils/cost_calculator.py): `app/utils/cost_calculator.py`
- [Quality metrics](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/quality_service.py): `app/services/quality_service.py`
- [Structured judge](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/judge_service.py): `app/services/judge_service.py`
- [Judge prompts](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/prompts/evaluation_prompts.py): `app/prompts/evaluation_prompts.py`
- [Guardrail checks](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/guardrail_service.py): `app/services/guardrail_service.py`
- [Regression comparison](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/services/regression_service.py): `app/services/regression_service.py`

### Persistence

- [Evaluation store](https://github.com/shreyasgaikwad-04/llm-evals-and-guardrails/blob/main/app/db/database.py): `app/db/database.py`

### Other components

- **Engineer**
- **OpenAI**
- **DeepEval**

## Connections

- Engineer → Streamlit dashboard: uses
- Streamlit dashboard → HTTP client: requests
- HTTP client → API routes: HTTP calls
- FastAPI application → API routes: mounts
- API routes → Dataset loader: loads records
- API routes → Evaluation runner: runs evaluation
- Evaluation runner → Guardrail checks: screens input
- Evaluation runner → LLM generation: generates answers
- LLM generation → OpenAI: requests completion
- LLM generation → Cost calculator: estimates cost
- Evaluation runner → Quality metrics: scores answers
- Quality metrics → DeepEval: computes metrics
- Quality metrics → Structured judge: requests judge score
- Structured judge → OpenAI: requests structured score
- Structured judge → Judge prompts: uses templates
- API routes → Guardrail checks: runs suite
- Guardrail checks → LLM generation: tests resistance
- API routes → Regression comparison: compares runs
- API routes → Evaluation store: reads and writes runs
- Streamlit dashboard → HTTP client: views run summaries