# aptsalt/rag-eval-engine architecture

> Architecture diagram of the GitHub repository aptsalt/rag-eval-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aptsalt/rag-eval-engine
- Repository: https://github.com/aptsalt/rag-eval-engine

## Overview

RAG Eval Engine ingests documents into searchable chunks, answers queries with hybrid retrieval and routed LLM generation, and evaluates answer quality with persisted metrics. Users can operate it through the six-page dashboard or MCP tools. The graph also preserves caching, adaptive retrieval, test-set evaluation, and the separate query and document stores. Edges reflect shown calls, SOURCE INDEX references, or README-documented interfaces; some subsystem internals are unsampled, so uncertain links are omitted.

Main entry points: [`src/evaluation/eval_pipeline.py`](https://github.com/aptsalt/rag-eval-engine/blob/main/src/evaluation/eval_pipeline.py) (Query Pipeline), [`src/mcp_server.py`](https://github.com/aptsalt/rag-eval-engine/blob/main/src/mcp_server.py) (MCP Server), [`src/retrieval/hybrid_ranker.py`](https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/hybrid_ranker.py) (Hybrid Ranker), [`src/main.py`](https://github.com/aptsalt/rag-eval-engine/blob/main/src/main.py) (FastAPI Gateway)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["User Interfaces"]
  node_dashboard["Dashboard Pages"]
  node_dashboard_api["Dashboard API Client<br/>[api.ts]"]
  node_mcp["MCP Server<br/>[mcp_server.py]"]
end

subgraph group_api["API and Workflow"]
  node_gateway["FastAPI Gateway<br/>[main.py]"]
  node_ingest_route["Ingest Routes<br/>[ingest.py]"]
  node_query_route["Query Routes<br/>[query.py]"]
  node_retrieve_route["Retrieve Routes<br/>[retrieve.py]"]
  node_evaluate_route["Evaluation Routes<br/>[evaluate.py]"]
end

subgraph group_ingestion["Ingestion and Retrieval"]
  node_loader["Document Loader<br/>[loader.py]"]
  node_chunker["Document Chunker<br/>[chunker.py]"]
  node_embedder["Embedding and Vector Store<br/>[embedder.py]"]
  node_hybrid["Hybrid Ranker<br/>[hybrid_ranker.py]"]
  node_vector["Vector Search<br/>[vector_search.py]"]
  node_sparse["Sparse Search<br/>[sparse_search.py]"]
end

subgraph group_generation["Generation and Evaluation"]
  node_pipeline["Query Pipeline<br/>[eval_pipeline.py]"]
  node_prompt["Prompt Builder<br/>[prompt_builder.py]"]
  node_llm["LLM Client<br/>[llm_client.py]"]
  node_metrics["Evaluation Metrics<br/>[metrics.py]"]
  node_cost["Cost Tracker<br/>[cost_tracker.py]"]
end

subgraph group_state["Persistence and Tuning"]
  node_autotune["Adaptive Retrieval<br/>[auto_tune.py]"]
  node_cache["Semantic Query Cache<br/>[query_cache.py]"]
  node_models[("Application Data Store<br/>[models.py]")]
end

node_user(("User"))
node_mcp_client(("MCP Client"))
node_qdrant[("Qdrant")]
node_sqlite[("SQLite")]
node_llm_providers{{"LLM Providers"}}

node_user -->|"uses"| node_dashboard
node_dashboard -->|"requests"| node_dashboard_api
node_dashboard_api -->|"REST and SSE"| node_gateway
node_mcp_client -->|"invokes tools"| node_mcp
node_gateway -->|"routes"| node_ingest_route
node_gateway -->|"routes"| node_query_route
node_gateway -->|"routes"| node_retrieve_route
node_gateway -->|"routes"| node_evaluate_route
node_ingest_route -->|"loads files"| node_loader
node_ingest_route -->|"chunks documents"| node_chunker
node_ingest_route -->|"embeds and stores"| node_embedder
node_ingest_route -->|"updates index"| node_sparse
node_embedder -->|"writes vectors"| node_qdrant
node_query_route -->|"runs query"| node_pipeline
node_query_route -->|"streams retrieval"| node_hybrid
node_query_route -->|"builds stream prompt"| node_prompt
node_query_route -->|"streams generation"| node_llm
node_retrieve_route -->|"searches"| node_hybrid
node_evaluate_route -->|"runs evaluations"| node_pipeline
node_hybrid -->|"combines results"| node_vector
node_hybrid -->|"combines results"| node_sparse
node_vector -->|"uses embeddings"| node_embedder
node_pipeline -->|"checks and stores"| node_cache
node_pipeline -->|"gets parameters"| node_autotune
node_pipeline -->|"retrieves context"| node_hybrid
node_pipeline -->|"builds prompt"| node_prompt
node_pipeline -->|"generates answer"| node_llm
node_pipeline -->|"scores answers"| node_metrics
node_pipeline -->|"logs query and eval"| node_models
node_pipeline -->|"tracks cost"| node_cost
node_cache -->|"embeds cache queries"| node_embedder
node_cache -->|"records cache stats"| node_models
node_models -->|"persists records"| node_sqlite
node_llm -->|"routes generation"| node_llm_providers
node_mcp -->|"exposes query tool"| node_pipeline
node_mcp -->|"exposes retrieval tool"| node_hybrid
node_mcp -->|"supports text ingest"| node_chunker
node_mcp -->|"supports text ingest"| node_embedder
node_mcp -->|"uses search index"| node_sparse

click node_dashboard "https://github.com/aptsalt/rag-eval-engine/tree/main/dashboard/src/app"
click node_dashboard_api "https://github.com/aptsalt/rag-eval-engine/blob/main/dashboard/src/lib/api.ts"
click node_mcp "https://github.com/aptsalt/rag-eval-engine/blob/main/src/mcp_server.py"
click node_gateway "https://github.com/aptsalt/rag-eval-engine/blob/main/src/main.py"
click node_ingest_route "https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/ingest.py"
click node_query_route "https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/query.py"
click node_retrieve_route "https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/retrieve.py"
click node_evaluate_route "https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/evaluate.py"
click node_loader "https://github.com/aptsalt/rag-eval-engine/blob/main/src/ingestion/loader.py"
click node_chunker "https://github.com/aptsalt/rag-eval-engine/blob/main/src/ingestion/chunker.py"
click node_embedder "https://github.com/aptsalt/rag-eval-engine/blob/main/src/ingestion/embedder.py"
click node_hybrid "https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/hybrid_ranker.py"
click node_vector "https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/vector_search.py"
click node_sparse "https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/sparse_search.py"
click node_pipeline "https://github.com/aptsalt/rag-eval-engine/blob/main/src/evaluation/eval_pipeline.py"
click node_prompt "https://github.com/aptsalt/rag-eval-engine/blob/main/src/generation/prompt_builder.py"
click node_llm "https://github.com/aptsalt/rag-eval-engine/blob/main/src/generation/llm_client.py"
click node_metrics "https://github.com/aptsalt/rag-eval-engine/blob/main/src/evaluation/metrics.py"
click node_cost "https://github.com/aptsalt/rag-eval-engine/blob/main/src/generation/cost_tracker.py"
click node_autotune "https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/auto_tune.py"
click node_cache "https://github.com/aptsalt/rag-eval-engine/blob/main/src/caching/query_cache.py"
click node_models "https://github.com/aptsalt/rag-eval-engine/blob/main/src/db/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_dashboard_api,node_mcp,node_user,node_mcp_client toneBlue
class node_gateway,node_ingest_route,node_query_route,node_retrieve_route,node_evaluate_route,node_qdrant,node_sqlite toneAmber
class node_loader,node_chunker,node_embedder,node_hybrid,node_vector,node_sparse toneMint
class node_pipeline,node_prompt,node_llm,node_metrics,node_cost toneRose
class node_autotune,node_cache,node_models,node_llm_providers toneIndigo
```

## Components

### User Interfaces

- [Dashboard Pages](https://github.com/aptsalt/rag-eval-engine/tree/main/dashboard/src/app): `dashboard/src/app`
- [Dashboard API Client](https://github.com/aptsalt/rag-eval-engine/blob/main/dashboard/src/lib/api.ts): `dashboard/src/lib/api.ts`
- [MCP Server](https://github.com/aptsalt/rag-eval-engine/blob/main/src/mcp_server.py): `src/mcp_server.py`

### API and Workflow

- [FastAPI Gateway](https://github.com/aptsalt/rag-eval-engine/blob/main/src/main.py): `src/main.py`
- [Ingest Routes](https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/ingest.py): `src/routes/ingest.py`
- [Query Routes](https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/query.py): `src/routes/query.py`
- [Retrieve Routes](https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/retrieve.py): `src/routes/retrieve.py`
- [Evaluation Routes](https://github.com/aptsalt/rag-eval-engine/blob/main/src/routes/evaluate.py): `src/routes/evaluate.py`

### Ingestion and Retrieval

- [Document Loader](https://github.com/aptsalt/rag-eval-engine/blob/main/src/ingestion/loader.py): `src/ingestion/loader.py`
- [Document Chunker](https://github.com/aptsalt/rag-eval-engine/blob/main/src/ingestion/chunker.py): `src/ingestion/chunker.py`
- [Embedding and Vector Store](https://github.com/aptsalt/rag-eval-engine/blob/main/src/ingestion/embedder.py): `src/ingestion/embedder.py`
- [Hybrid Ranker](https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/hybrid_ranker.py): `src/retrieval/hybrid_ranker.py`
- [Vector Search](https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/vector_search.py): `src/retrieval/vector_search.py`
- [Sparse Search](https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/sparse_search.py): `src/retrieval/sparse_search.py`

### Generation and Evaluation

- [Query Pipeline](https://github.com/aptsalt/rag-eval-engine/blob/main/src/evaluation/eval_pipeline.py): `src/evaluation/eval_pipeline.py`
- [Prompt Builder](https://github.com/aptsalt/rag-eval-engine/blob/main/src/generation/prompt_builder.py): `src/generation/prompt_builder.py`
- [LLM Client](https://github.com/aptsalt/rag-eval-engine/blob/main/src/generation/llm_client.py): `src/generation/llm_client.py`
- [Evaluation Metrics](https://github.com/aptsalt/rag-eval-engine/blob/main/src/evaluation/metrics.py): `src/evaluation/metrics.py`
- [Cost Tracker](https://github.com/aptsalt/rag-eval-engine/blob/main/src/generation/cost_tracker.py): `src/generation/cost_tracker.py`

### Persistence and Tuning

- [Adaptive Retrieval](https://github.com/aptsalt/rag-eval-engine/blob/main/src/retrieval/auto_tune.py): `src/retrieval/auto_tune.py`
- [Semantic Query Cache](https://github.com/aptsalt/rag-eval-engine/blob/main/src/caching/query_cache.py): `src/caching/query_cache.py`
- [Application Data Store](https://github.com/aptsalt/rag-eval-engine/blob/main/src/db/models.py): `src/db/models.py`

### Other components

- **User**
- **MCP Client**
- **Qdrant**
- **SQLite**
- **LLM Providers**

## Connections

- User → Dashboard Pages: uses (evidence: `README.md`)
- Dashboard Pages → Dashboard API Client: requests (evidence: `README.md`)
- Dashboard API Client → FastAPI Gateway: REST and SSE (evidence: `README.md`)
- MCP Client → MCP Server: invokes tools (evidence: `README.md`)
- FastAPI Gateway → Ingest Routes: routes (evidence: `src/main.py`)
- FastAPI Gateway → Query Routes: routes (evidence: `src/main.py`)
- FastAPI Gateway → Retrieve Routes: routes (evidence: `src/main.py`)
- FastAPI Gateway → Evaluation Routes: routes (evidence: `src/main.py`)
- Ingest Routes → Document Loader: loads files (evidence: `src/routes/ingest.py`)
- Ingest Routes → Document Chunker: chunks documents (evidence: `src/routes/ingest.py`)
- Ingest Routes → Embedding and Vector Store: embeds and stores (evidence: `src/routes/ingest.py`)
- Ingest Routes → Sparse Search: updates index (evidence: `src/routes/ingest.py`)
- Embedding and Vector Store → Qdrant: writes vectors (evidence: `src/ingestion/embedder.py`)
- Query Routes → Query Pipeline: runs query (evidence: `src/routes/query.py`)
- Query Routes → Hybrid Ranker: streams retrieval (evidence: `src/routes/query.py`)
- Query Routes → Prompt Builder: builds stream prompt (evidence: `src/routes/query.py`)
- Query Routes → LLM Client: streams generation (evidence: `src/routes/query.py`)
- Retrieve Routes → Hybrid Ranker: searches (evidence: `src/routes/retrieve.py`)
- Evaluation Routes → Query Pipeline: runs evaluations (evidence: `src/routes/evaluate.py`)
- Hybrid Ranker → Vector Search: combines results (evidence: `src/retrieval/hybrid_ranker.py`)
- Hybrid Ranker → Sparse Search: combines results (evidence: `src/retrieval/hybrid_ranker.py`)
- Vector Search → Embedding and Vector Store: uses embeddings (evidence: `src/retrieval/vector_search.py`)
- Query Pipeline → Semantic Query Cache: checks and stores (evidence: `src/evaluation/eval_pipeline.py`)
- Query Pipeline → Adaptive Retrieval: gets parameters (evidence: `src/evaluation/eval_pipeline.py`)
- Query Pipeline → Hybrid Ranker: retrieves context (evidence: `src/evaluation/eval_pipeline.py`)
- Query Pipeline → Prompt Builder: builds prompt (evidence: `src/evaluation/eval_pipeline.py`)
- Query Pipeline → LLM Client: generates answer (evidence: `src/evaluation/eval_pipeline.py`)
- Query Pipeline → Evaluation Metrics: scores answers (evidence: `src/evaluation/eval_pipeline.py`)
- Query Pipeline → Application Data Store: logs query and eval (evidence: `src/evaluation/eval_pipeline.py`)
- Query Pipeline → Cost Tracker: tracks cost (evidence: `src/evaluation/eval_pipeline.py`)
- Semantic Query Cache → Embedding and Vector Store: embeds cache queries (evidence: `src/caching/query_cache.py`)
- Semantic Query Cache → Application Data Store: records cache stats (evidence: `src/caching/query_cache.py`)
- Application Data Store → SQLite: persists records (evidence: `src/db/models.py`)
- LLM Client → LLM Providers: routes generation (evidence: `src/generation/llm_client.py`)
- MCP Server → Query Pipeline: exposes query tool (evidence: `src/mcp_server.py`)
- MCP Server → Hybrid Ranker: exposes retrieval tool (evidence: `src/mcp_server.py`)
- MCP Server → Document Chunker: supports text ingest (evidence: `src/mcp_server.py`)
- MCP Server → Embedding and Vector Store: supports text ingest (evidence: `src/mcp_server.py`)
- MCP Server → Sparse Search: uses search index (evidence: `src/mcp_server.py`)
