How shreyasgaikwad-04/llm-evals-and-guardrails works
The platform takes a dataset and model configuration through a Streamlit dashboard or API, validates records, generates model answers, evaluates quality and guardrails, and stores evaluation runs for review and regression comparison. The graph separates the user interface, API orchestration, evaluation capabilities, and persistence. Optional provider-backed evaluation is distinguished from deterministic fallback behavior. The README documents dashboard and metric capabilities; unsampled dashboard internals and database models limit detail about their implementation.
Main entry points: routes.py API routes · evaluation_service.py Evaluation runner · llm_service.py LLM generation · streamlit_app.py Streamlit dashboard
Components 17
User interface
- Streamlit dashboard
dashboard/streamlit_app.py - HTTP client
dashboard/api_client.py
API orchestration
- API routes
app/api/routes.py - FastAPI application
app/main.py
Evaluation capabilities
- Dataset loader
app/utils/dataset_loader.py - Evaluation runner
app/services/evaluation_service.py - LLM generation
app/services/llm_service.py - Cost calculator
app/utils/cost_calculator.py - Quality metrics
app/services/quality_service.py - Structured judge
app/services/judge_service.py - Judge prompts
app/prompts/evaluation_prompts.py - Guardrail checks
app/services/guardrail_service.py - Regression comparison
app/services/regression_service.py
Persistence
- Evaluation store
app/db/database.py
Other components
- Engineer
- OpenAI
- DeepEval
Connections 20
- Engineer → Streamlit dashboard: uses
- Streamlit dashboard → HTTP client: requests
- HTTP client → API routes: HTTP calls
- FastAPI application → API routes: mounts
- API routes → Dataset loader: loads records
- API routes → Evaluation runner: runs evaluation
- Evaluation runner → Guardrail checks: screens input
- Evaluation runner → LLM generation: generates answers
- LLM generation → OpenAI: requests completion
- LLM generation → Cost calculator: estimates cost
- Evaluation runner → Quality metrics: scores answers
- Quality metrics → DeepEval: computes metrics
- Quality metrics → Structured judge: requests judge score
- Structured judge → OpenAI: requests structured score
- Structured judge → Judge prompts: uses templates
- API routes → Guardrail checks: runs suite
- Guardrail checks → LLM generation: tests resistance
- API routes → Regression comparison: compares runs
- API routes → Evaluation store: reads and writes runs
- Streamlit dashboard → HTTP client: views run summaries