How sumeet-giri/ai-response-quality-evaluator-agent works
This is a multi-agent AI response-quality evaluator with a Streamlit interface and FastAPI backend. Users submit a question, response, and optional reference; evaluation scores relevance, accuracy, hallucination risk, and completeness, then produces a verdict and report. The app also supports CSV batch evaluation and historical dashboards. The graph includes the documented retrieval fallback and validation capabilities. Edges rely on excerpted code or SOURCE INDEX references; unsampled implementation details are not inferred.
Main entry points: orchestrator.py Evaluation Pipeline · evaluate.py Evaluation Routes · single_evaluation.py Single Evaluation · benchmark_validation.py Batch Evaluation
Components 32
Streamlit Interface
- Page Router
frontend/app.py - Single Evaluation
frontend/pages_content/single_evaluation.py - Batch Evaluation
frontend/pages_content/benchmark_validation.py - History Dashboard
frontend/pages_content/dashboard.py - Backend Client
frontend/utils/api_client.py - Batch Processor
frontend/utils/batch_processor.py - CSV Parser
frontend/utils/csv_parser.py - PDF Reports
frontend/utils/pdf_export.py - Score Cards
frontend/components/result_cards.py - Score Charts
frontend/components/charts.py - Batch Charts
frontend/components/batch_charts.py - UI Metrics
frontend/components/badges.py - Pipeline View
frontend/components/pipeline_viz.py
Evaluation Backend
- FastAPI App
backend/app/main.py - Evaluation Routes
backend/app/api/evaluate.py - Evaluation Pipeline
backend/app/orchestration/orchestrator.py
Scoring Agents
- Relevance Scorer
backend/app/agents/relevance_agent.py - Accuracy Scorer
backend/app/agents/accuracy_agent.py - Hallucination Scorer
backend/app/agents/hallucination_agent.py - Completeness Scorer
backend/app/agents/completeness_agent.py - Verdict Scorer
backend/app/agents/verdict_agent.py
Reference Retrieval
- Text Embedder
backend/app/services/embedder.py - Similarity
backend/app/services/similarity.py - Reference Retriever
backend/app/services/retriever.py - Knowledge Collection
backend/app/services/chroma_manager.py
History and Validation
- History Routes
backend/app/api/history.py - Evaluation History
backend/app/services/history_store.py - Validation Route
backend/app/api/validation.py - Benchmark Validator
backend/app/validation/validator.py - Benchmark Cases
backend/app/validation/benchmark_cases.py
Other components
- Evaluator
- Embedding Model
Connections 48
- Evaluator → Page Router: uses
frontend/app.py - Page Router → Single Evaluation: routes to
frontend/app.py - Page Router → Batch Evaluation: routes to
frontend/app.py - Page Router → History Dashboard: routes to
frontend/app.py - Single Evaluation → Backend Client: requests scores
frontend/pages_content/single_evaluation.py - Single Evaluation → Score Cards: renders
frontend/pages_content/single_evaluation.py - Single Evaluation → Score Charts: renders
frontend/pages_content/single_evaluation.py - Single Evaluation → Pipeline View: renders
frontend/pages_content/single_evaluation.py - Single Evaluation → PDF Reports: exports report
frontend/pages_content/single_evaluation.py - Batch Evaluation → CSV Parser: parses upload
frontend/pages_content/benchmark_validation.py - Batch Evaluation → Batch Processor: runs rows
frontend/pages_content/benchmark_validation.py - Batch Evaluation → PDF Reports: exports report
frontend/pages_content/benchmark_validation.py - Batch Evaluation → Batch Charts: renders analytics
frontend/pages_content/benchmark_validation.py - Batch Evaluation → UI Metrics: renders metrics
frontend/pages_content/benchmark_validation.py - History Dashboard → Backend Client: loads history
frontend/pages_content/dashboard.py - History Dashboard → Score Charts: renders charts
frontend/pages_content/dashboard.py - History Dashboard → Batch Charts: renders charts
frontend/pages_content/dashboard.py - History Dashboard → UI Metrics: renders metrics
frontend/pages_content/dashboard.py - Backend Client → Evaluation Routes: calls evaluation
frontend/utils/api_client.py - Backend Client → History Routes: fetches history
frontend/utils/api_client.py - FastAPI App → Evaluation Routes: mounts routes
backend/app/main.py - FastAPI App → History Routes: mounts routes
backend/app/main.py - FastAPI App → Validation Route: mounts routes
backend/app/main.py - Evaluation Routes → Evaluation Pipeline: runs pipeline
backend/app/api/evaluate.py - Evaluation Routes → Relevance Scorer: scores relevance
backend/app/api/evaluate.py - Evaluation Routes → Accuracy Scorer: scores accuracy
backend/app/api/evaluate.py - Evaluation Routes → Hallucination Scorer: scores hallucination
backend/app/api/evaluate.py - Evaluation Routes → Completeness Scorer: scores completeness
backend/app/api/evaluate.py - Evaluation Pipeline → Reference Retriever: retrieves evidence
backend/app/orchestration/orchestrator.py - Evaluation Pipeline → Relevance Scorer: scores relevance
backend/app/orchestration/orchestrator.py - Evaluation Pipeline → Accuracy Scorer: scores accuracy
backend/app/orchestration/orchestrator.py - Evaluation Pipeline → Hallucination Scorer: scores claims
backend/app/orchestration/orchestrator.py - Evaluation Pipeline → Completeness Scorer: scores coverage
backend/app/orchestration/orchestrator.py - Evaluation Pipeline → Verdict Scorer: builds verdict
backend/app/orchestration/orchestrator.py - Evaluation Pipeline → Evaluation History: records results
backend/app/orchestration/orchestrator.py - Relevance Scorer → Text Embedder: embeds text
backend/app/agents/relevance_agent.py - Relevance Scorer → Similarity: compares vectors
backend/app/agents/relevance_agent.py - Accuracy Scorer → Text Embedder: embeds text
backend/app/agents/accuracy_agent.py - Accuracy Scorer → Similarity: compares vectors
backend/app/agents/accuracy_agent.py - Hallucination Scorer → Text Embedder: embeds claims
backend/app/agents/hallucination_agent.py - Hallucination Scorer → Similarity: compares claims
backend/app/agents/hallucination_agent.py - Reference Retriever → Text Embedder: embeds query
backend/app/services/retriever.py - Reference Retriever → Knowledge Collection: queries collection
backend/app/services/retriever.py - Text Embedder → Embedding Model: uses
- History Routes → Evaluation History: reads history
backend/app/api/history.py - Validation Route → Benchmark Validator: runs validation
backend/app/api/validation.py - Benchmark Validator → Relevance Scorer: evaluates relevance
backend/app/validation/validator.py - Benchmark Validator → Accuracy Scorer: evaluates accuracy
backend/app/validation/validator.py