How aptsalt/rag-eval-engine works
RAG Eval Engine ingests documents into searchable chunks, answers queries with hybrid retrieval and routed LLM generation, and evaluates answer quality with persisted metrics. Users can operate it through the six-page dashboard or MCP tools. The graph also preserves caching, adaptive retrieval, test-set evaluation, and the separate query and document stores. Edges reflect shown calls, SOURCE INDEX references, or README-documented interfaces; some subsystem internals are unsampled, so uncertain links are omitted.
Main entry points: eval_pipeline.py Query Pipeline · mcp_server.py MCP Server · hybrid_ranker.py Hybrid Ranker · main.py FastAPI Gateway
Components 27
User Interfaces
- Dashboard Pages
dashboard/src/app - Dashboard API Client
dashboard/src/lib/api.ts - MCP Server
src/mcp_server.py
API and Workflow
- FastAPI Gateway
src/main.py - Ingest Routes
src/routes/ingest.py - Query Routes
src/routes/query.py - Retrieve Routes
src/routes/retrieve.py - Evaluation Routes
src/routes/evaluate.py
Ingestion and Retrieval
- Document Loader
src/ingestion/loader.py - Document Chunker
src/ingestion/chunker.py - Embedding and Vector Store
src/ingestion/embedder.py - Hybrid Ranker
src/retrieval/hybrid_ranker.py - Vector Search
src/retrieval/vector_search.py - Sparse Search
src/retrieval/sparse_search.py
Generation and Evaluation
- Query Pipeline
src/evaluation/eval_pipeline.py - Prompt Builder
src/generation/prompt_builder.py - LLM Client
src/generation/llm_client.py - Evaluation Metrics
src/evaluation/metrics.py - Cost Tracker
src/generation/cost_tracker.py
Persistence and Tuning
- Adaptive Retrieval
src/retrieval/auto_tune.py - Semantic Query Cache
src/caching/query_cache.py - Application Data Store
src/db/models.py
Other components
- User
- MCP Client
- Qdrant
- SQLite
- LLM Providers
Connections 39
- User → Dashboard Pages: uses
README.md - Dashboard Pages → Dashboard API Client: requests
README.md - Dashboard API Client → FastAPI Gateway: REST and SSE
README.md - MCP Client → MCP Server: invokes tools
README.md - FastAPI Gateway → Ingest Routes: routes
src/main.py - FastAPI Gateway → Query Routes: routes
src/main.py - FastAPI Gateway → Retrieve Routes: routes
src/main.py - FastAPI Gateway → Evaluation Routes: routes
src/main.py - Ingest Routes → Document Loader: loads files
src/routes/ingest.py - Ingest Routes → Document Chunker: chunks documents
src/routes/ingest.py - Ingest Routes → Embedding and Vector Store: embeds and stores
src/routes/ingest.py - Ingest Routes → Sparse Search: updates index
src/routes/ingest.py - Embedding and Vector Store → Qdrant: writes vectors
src/ingestion/embedder.py - Query Routes → Query Pipeline: runs query
src/routes/query.py - Query Routes → Hybrid Ranker: streams retrieval
src/routes/query.py - Query Routes → Prompt Builder: builds stream prompt
src/routes/query.py - Query Routes → LLM Client: streams generation
src/routes/query.py - Retrieve Routes → Hybrid Ranker: searches
src/routes/retrieve.py - Evaluation Routes → Query Pipeline: runs evaluations
src/routes/evaluate.py - Hybrid Ranker → Vector Search: combines results
src/retrieval/hybrid_ranker.py - Hybrid Ranker → Sparse Search: combines results
src/retrieval/hybrid_ranker.py - Vector Search → Embedding and Vector Store: uses embeddings
src/retrieval/vector_search.py - Query Pipeline → Semantic Query Cache: checks and stores
src/evaluation/eval_pipeline.py - Query Pipeline → Adaptive Retrieval: gets parameters
src/evaluation/eval_pipeline.py - Query Pipeline → Hybrid Ranker: retrieves context
src/evaluation/eval_pipeline.py - Query Pipeline → Prompt Builder: builds prompt
src/evaluation/eval_pipeline.py - Query Pipeline → LLM Client: generates answer
src/evaluation/eval_pipeline.py - Query Pipeline → Evaluation Metrics: scores answers
src/evaluation/eval_pipeline.py - Query Pipeline → Application Data Store: logs query and eval
src/evaluation/eval_pipeline.py - Query Pipeline → Cost Tracker: tracks cost
src/evaluation/eval_pipeline.py - Semantic Query Cache → Embedding and Vector Store: embeds cache queries
src/caching/query_cache.py - Semantic Query Cache → Application Data Store: records cache stats
src/caching/query_cache.py - Application Data Store → SQLite: persists records
src/db/models.py - LLM Client → LLM Providers: routes generation
src/generation/llm_client.py - MCP Server → Query Pipeline: exposes query tool
src/mcp_server.py - MCP Server → Hybrid Ranker: exposes retrieval tool
src/mcp_server.py - MCP Server → Document Chunker: supports text ingest
src/mcp_server.py - MCP Server → Embedding and Vector Store: supports text ingest
src/mcp_server.py - MCP Server → Sparse Search: uses search index
src/mcp_server.py