How xishhh/adaptive-rag-evaluation-system works
1. Overall shape
This repo is a two-part application: a FastAPI backend that implements adaptive RAG and document ingestion, and a separate Vite/React UI that provides chat, upload, and source inspection. The backend owns all retrieval, LLM orchestration, vector storage, and evaluation logic; the UI is a thin client over HTTP and SSE.
2. Request entry points
The backend entrypoint is app/main.py, which wires together the API routers under app/api/*. The exposed surface is centered on health, upload/status, query, streaming query, evaluation, and session cleanup endpoints. app/api/dependencies.py likely provides shared service instances and configuration injection across handlers.
3. Query serving path
User questions flow from /query or /query/stream into AdaptiveRAGService and RAGService. Before retrieval, the system classifies the query via QueryClassifier into direct LLM answers versus knowledge-grounded answers. Knowledge queries then enter the retrieval pipeline; direct queries bypass vector search and go to the LLM path.
4. Adaptive retrieval loop
The retrieval subsystem in app/retrievers/* is the core differentiator. HybridRetriever performs document lookup against the vector store, ConfidenceEvaluator scores whether the retrieved context is good enough, and QueryRewriter reformulates low-confidence queries for another retrieval pass. This creates an iterative loop: retrieve, score, possibly rewrite, then retrieve again before answering.
5. LLM orchestration and fallback
app/utils/llm_factory.py and the services layer manage model selection and fallback behavior. The README indicates a primary-plus-fallback model strategy with retry/backoff on API errors or rate limits. This means the LLM is not a single hard dependency; it is a resilient external boundary with graceful degradation.
6. Document ingestion pipeline
Uploads go through app/api/upload.py into IngestionService. The ingestion path uses loaders.py for format-specific parsing and chunker.py for splitting documents into retrievable segments. Processing is asynchronous: upload returns a job id immediately, while the file is indexed in the background and tracked via job status polling.
7. Vector storage boundary
app/vectorstore/chroma_manager.py encapsulates ChromaDB, which is the persistence layer for embeddings and document chunks. The README’s idempotent re-upload behavior implies the store supports replacement semantics keyed by document identity, so the vector store is not just append-only. This is the main durable data store for RAG context.
8. Session and lifecycle management
SessionService and the /session endpoint provide a hard reset boundary for the system. Clearing a session removes uploaded files, ingestion jobs, and Chroma data, and the README notes auto-cleanup on server shutdown. This makes the app effectively stateful per session while still supporting full teardown.
9. Answer composition and citations
CitationService is the presentation-layer bridge between retrieved chunks and the final answer. It likely attaches source references to responses so the UI can render provenance in a side panel. This is important architecturally because the retrieval layer’s output is not just context for the model; it is also exposed as user-facing evidence.
10. Evaluation subsystem
The app/evaluators/* package is a separate offline/ops-oriented pipeline. eval_dataset_generator.py builds evaluation sets using real retrieval contexts and system outputs, ragas_evaluator.py runs RAGAS metrics, and metrics_tracker.py stores or summarizes results. This subsystem is invoked through /evaluate and /evaluate/results, and is distinct from live query serving.
11. Data models and contracts
app/models/requests.py, responses.py, and documents.py define the API and internal schema boundary. These Pydantic models normalize documents, upload jobs, query inputs, streaming responses, and evaluation payloads so the rest of the system can remain service-oriented rather than ad hoc JSON-driven.
12. Frontend interaction model
The ui/src tree contains a chat-centric interface with file upload, sidebar, and source panel components. ui/src/api/client.ts and sse.ts show two backend interaction modes: standard request/response for uploads and queries, and SSE for token streaming. ChatContext and DocumentContext indicate the UI maintains conversational state and document/job state separately.
13. Operational concerns and verification
The repo includes a Dockerfile for containerized deployment and pytest coverage for health, ingestion, and vector store behavior. That suggests the most important correctness boundaries are service availability, ingestion/indexing behavior, and persistence semantics rather than UI rendering. The tests are focused where architectural regressions would be most costly.
14. Key architecture takeaway
The system is organized around one central loop: ingest documents into ChromaDB, classify incoming questions, retrieve and score context adaptively, then answer with an LLM and citations. The main boundaries are API vs services, services vs vector store, and backend vs UI; the adaptive retrieval and async ingestion paths are the two distinctive flows that should be highlighted in any architecture diagram.
Main entry points: ChatPage.tsx Chat page · main.py API app · ingestion_service.py Ingestion · rag_service.py RAG service
Components 31
Backend
FastAPI app for ingestion, adaptive RAG, and evaluation.
- API app · Wires routers, dependencies, and app startup.
app/main.py - Health · Liveness and readiness surface.
app/api/health.py - Upload API · Starts ingestion jobs for uploaded documents.
app/api/upload.py - Query API · Handles synchronous and streaming question answering.
app/api/query.py - Eval API · Triggers evaluation runs and exposes results.
app/api/evaluate.py - Session API · Resets session state and cleanup boundaries.
app/api/session.py - Dependencies · Shared service and config injection for API handlers.
app/api/dependencies.py - Ingestion · Coordinates upload processing and background indexing.
app/ingestion/ingestion_service.py - Loaders · Format-specific document loading and normalization.
app/ingestion/loaders.py - Chunker · Splits documents into retrievable segments.
app/ingestion/chunker.py - Ingestion tracker · Tracks asynchronous upload and indexing status.
app/services/ingestion_tracker.py - Adaptive RAG · Runs the retrieve-score-rewrite-answer loop.
app/services/adaptive_rag_service.py - RAG service · Builds grounded answers from retrieved context.
app/services/rag_service.py - Query classifier · Splits direct questions from knowledge-grounded ones.
app/retrievers/query_classifier.py - Hybrid retriever · Fetches candidate chunks from vector storage.
app/retrievers/hybrid_retriever.py - Confidence eval · Scores whether retrieved context is sufficient.
app/retrievers/confidence_evaluator.py - Query rewriter · Rephrases weak queries for another retrieval pass.
app/retrievers/query_rewriter.py - Citations · Attaches source references to final answers.
app/services/citation_service.py - Eval dataset · Builds datasets from real retrieval and answer traces.
app/evaluators/eval_dataset_generator.py - RAGAS eval · Computes RAG evaluation metrics.
app/evaluators/ragas_evaluator.py - Metrics tracker · Collects and summarizes evaluation outputs.
app/evaluators/metrics_tracker.py
UI
Vite/React client for chat, uploads, and source inspection.
- UI app · Bootstraps the client application.
ui/src/App.tsx - Chat page · Main conversational workspace with upload and sources.
ui/src/pages/ChatPage.tsx - Chat state · Manages conversation and streaming answer state.
ui/src/context/ChatContext.tsx - Document state · Tracks uploads, jobs, and source documents.
ui/src/context/DocumentContext.tsx - API client · Handles standard backend requests.
ui/src/api/client.ts - SSE client · Consumes token streaming responses.
ui/src/api/sse.ts - Uploader · Starts document upload flows from the UI.
ui/src/components/FileUploader.tsx - Source panel · Displays retrieved sources and citations.
ui/src/components/SourcePanel.tsx
Ops
Shared runtime, storage, and deployment boundaries.
- LLM factory · Selects primary and fallback model clients.
app/utils/llm_factory.py - Chroma store · Persists embeddings and document chunks.
app/vectorstore/chroma_manager.py
Connections 39
- API app → Health: mounts · Registers health endpoints.
- API app → Upload API: mounts · Registers upload endpoints.
- API app → Query API: mounts · Registers query endpoints.
- API app → Eval API: mounts · Registers evaluation endpoints.
- API app → Session API: mounts · Registers session endpoints.
- Upload API → Dependencies: uses · Resolves shared ingestion services.
- Query API → Dependencies: uses · Resolves shared query services.
- Eval API → Dependencies: uses · Resolves evaluation services.
- Session API → Dependencies: uses · Resolves session cleanup services.
- Upload API → Ingestion: starts · Creates background ingestion jobs.
- Ingestion → Loaders: parses · Loads uploaded files into documents.
- Ingestion → Chunker: splits · Chunks content for retrieval.
- Ingestion → Ingestion tracker: tracks · Updates job progress and status.
- Ingestion → Chroma store: indexes · Persists chunks and embeddings.
- Query API → Adaptive RAG: calls · Routes questions into the answer pipeline.
- Adaptive RAG → Query classifier: routes · Decides direct vs grounded answering.
- Adaptive RAG → RAG service: delegates · Hands off grounded answering flow.
- RAG service → Hybrid retriever: retrieves · Fetches relevant context from storage.
- Hybrid retriever → Chroma store: queries · Searches indexed chunks.
- RAG service → Confidence eval: scores · Evaluates retrieval quality.
- Confidence eval → Query rewriter: triggers · Requests another retrieval attempt.
- Query rewriter → Hybrid retriever: rewrites for · Feeds reformulated query back to retrieval.
- RAG service → LLM factory: uses · Obtains primary or fallback model clients.
- RAG service → Citations: enriches · Adds source references to answers.
- Eval API → Eval dataset: generates · Builds evaluation inputs from system traces.
- Eval API → RAGAS eval: runs · Executes RAG evaluation metrics.
- RAGAS eval → Metrics tracker: stores · Persists evaluation results.
- UI app → Chat page: renders · Hosts the main chat experience.
- Chat page → Chat state: uses · Reads and updates conversation state.
- Chat page → Document state: uses · Reads upload and source state.
- Chat page → API client: requests · Sends normal API calls.
- Chat page → SSE client: streams · Consumes streaming answers.
- Chat page → Uploader: uploads · Starts document ingestion from the UI.
- Chat page → Source panel: shows · Displays citations and retrieved evidence.
- API client → Upload API: calls · Submits uploads and polls job status.
- API client → Query API: calls · Sends chat questions.
- API client → Eval API: calls · Triggers evaluation actions.
- API client → Session API: calls · Requests session reset.
- SSE client → Query API: subscribes · Consumes streamed query responses.