How jerry-ai-dev/modular-rag-mcp-server works
1. What this repository is
This is a Python-based modular RAG system exposed as an MCP server. The runtime product is not just retrieval: it includes ingestion, hybrid search, reranking, response assembly with citations/multimodal support, observability, evaluation, and a Streamlit dashboard.
2. Top-level runtime entrypoints
The main operational entrypoints are main.py and the scripts under scripts/: ingest.py, query.py, evaluate.py, and start_dashboard.py. These are thin orchestration layers that connect configuration, pipeline components, and the service surfaces.
3. Core architectural boundary: src/libs vs src/core vs src/ingestion
src/libs contains the pluggable provider/adaptor implementations and factories: LLMs, embeddings, rerankers, splitters, vector store, loader, and evaluator backends. src/ingestion and src/core contain the application workflows built on top of those adapters. This separation is the key design boundary: backends can be swapped without changing the main pipelines.
4. Ingestion pipeline: document to indexed knowledge
The ingestion flow is centered in src/ingestion/pipeline.py and document_manager.py. Documents are loaded via loader implementations such as pdf_loader.py, checked by file_integrity.py, chunked by document_chunker.py, optionally refined/enriched/captioned through transforms, embedded in batches, and finally written to storage. The storage side fans out into a vector upsert path, a BM25 indexer, and image storage for multimodal assets.
5. Multimodal handling
The repository treats images as first-class knowledge content. image_captioner.py uses a vision LLM backend to generate text descriptions, so images can be folded back into the text-centric RAG pipeline. This lets the retrieval layer search across documents with embedded visual context without needing a separate image QA subsystem.
6. Retrieval architecture: hybrid search
Query-time retrieval is implemented in src/core/query_engine. The flow is: query processing → dense retrieval → sparse retrieval → fusion → optional reranking. Dense search uses embeddings and the vector store; sparse search uses BM25; fusion.py combines them with reciprocal rank fusion; rerankers then refine the top candidates. This is the central relevance architecture of the system.
7. Response assembly
Retrieved chunks are turned into final answers in src/core/response. response_builder.py coordinates answer construction, citation_generator.py attaches source references, and multimodal_assembler.py can merge textual and image-derived context. This is where the system turns retrieval results into an assistant-facing response rather than just returning documents.
8. MCP server layer
src/mcp_server wraps the RAG capability as standardized MCP tools. The tools exposed in tools/ include query_knowledge_hub, list_collections, and get_document_summary. protocol_handler.py and server.py handle the protocol boundary, allowing external MCP clients such as AI assistants to call into the knowledge system.
9. Observability and traces
The system is designed to be white-box observable. src/core/trace defines trace context and collection for ingestion/query lifecycles, and the dashboard services in src/observability/dashboard/services read those traces and associated data. The dashboard pages provide views for overview, data browsing, ingestion management, ingestion traces, query traces, and evaluation results.
10. Evaluation and regression testing
Evaluation is a first-class subsystem under src/observability/evaluation. eval_runner.py, composite_evaluator.py, ragas_evaluator.py, and custom_evaluator.py support both framework-based and custom scoring. The repo also includes a golden test set in tests/fixtures/ and end-to-end/integration tests to validate ingestion, retrieval recall, MCP behavior, and dashboard smoke tests.
11. Configuration and prompt assets
src/core/settings.py and config/settings.yaml centralize runtime configuration. config/prompts/ stores prompt templates for chunk refinement, metadata enrichment, image captioning, and reranking. This keeps prompt behavior and backend selection outside business logic, reinforcing the pluggable design.
12. External services and backend options
The repo is built to work with multiple provider families: OpenAI, Azure OpenAI, Ollama, and DeepSeek for LLM/vision tasks; ChromaDB as the vector store; BM25 as sparse retrieval. The factories in src/libs/*_factory.py are the switch points that let the same architecture run against different local or cloud backends.
13. Testing and validation structure
The tests/ tree mirrors the runtime boundaries: unit tests validate isolated adapters and core logic, integration tests cover real provider and storage interactions, and e2e tests exercise the full ingestion/query/dashboard/MCP flows. This mirrors the architecture well and is important for any diagram because it shows which boundaries are meant to be stable integration points.
14. Developer tooling is separate from runtime
The .claude/ and .github/skills/ content is AI-assisted development tooling: skills for auto-coding, QA, packaging, setup, and related workflows. These are not runtime components of the RAG server, but they do explain how the project is intended to be built, tested, and packaged.
Main entry points: server.py MCP server · hybrid_search.py Query engine · response_builder.py Response · pipeline.py Pipeline
Components 34
Entry points
Runtime scripts and app launchers
Ingestion
Document-to-index pipeline
- Pipeline · Coordinates document ingestion
src/ingestion/pipeline.py - Docs · Loads, validates, and manages source documents
src/ingestion/document_manager.py - Chunking · Splits documents into chunks
src/ingestion/chunking/document_chunker.py - Transforms · Refines chunks, enriches metadata, captions images
src/ingestion/transform/base_transform.py - Embeddings · Batches and encodes dense and sparse representations
src/ingestion/embedding/batch_processor.py - Storage · Writes vectors, BM25, and image assets
src/ingestion/storage/vector_upserter.py
Retrieval and response
Query-time relevance and answer assembly
- Query engine · Coordinates query processing and retrieval
src/core/query_engine/hybrid_search.py - Query processing · Prepares the user query for retrieval
src/core/query_engine/query_processor.py - Hybrid search · Runs dense and sparse retrieval before fusion
src/core/query_engine/hybrid_search.py - Fusion · Combines candidate lists with RRF
src/core/query_engine/fusion.py - Reranker · Refines top candidates
src/core/query_engine/reranker.py - Response · Builds final assistant response
src/core/response/response_builder.py - Citations · Attaches source references
src/core/response/citation_generator.py - Multimodal · Merges text and image-derived context
src/core/response/multimodal_assembler.py
Providers
Pluggable backend adapters and factories
- LLMs · Selects text and vision model backends
src/libs/llm/llm_factory.py - Embeddings backends · Selects dense embedding backends
src/libs/embedding/embedding_factory.py - Rerankers · Selects reranking backends
src/libs/reranker/reranker_factory.py - Vector store · Selects the vector database backend
src/libs/vector_store/vector_store_factory.py
MCP server
Protocol boundary and exposed tools
- MCP server · Exposes RAG capability over MCP
src/mcp_server/server.py - Protocol · Handles MCP request/response flow
src/mcp_server/protocol_handler.py - Ask hub · Knowledge query tool
src/mcp_server/tools/query_knowledge_hub.py - List collections · Enumerates available collections
src/mcp_server/tools/list_collections.py - Doc summary · Returns a document summary
src/mcp_server/tools/get_document_summary.py
Observability
Tracing, dashboards, and evaluation
- Tracing · Collects ingestion and query traces
src/core/trace/trace_collector.py - Dashboard · Streamlit dashboard shell
src/observability/dashboard/app.py - Trace service · Reads and serves trace data
src/observability/dashboard/services/trace_service.py - Evaluation · Runs composite and RAGAS evaluation
src/observability/evaluation/eval_runner.py
Connections 40
- main.py → MCP server: serves · Launches the MCP-facing runtime
- main.py → Pipeline: orchestrates · Can initialize ingestion-related services
- Ingest → Pipeline: runs · Triggers document ingestion
- Query → Query engine: runs · Executes retrieval and answering
- Evaluate → Evaluation: runs · Launches evaluation jobs
- Dashboard → Dashboard: starts · Opens observability UI
- Pipeline → Docs: loads · Coordinates source document handling
- Docs → Chunking: chunks · Breaks documents into retrievable units
- Docs → Transforms: enriches · Applies refinement and metadata/image transforms
- Chunking → Embeddings: feeds · Produces text units for encoding
- Transforms → LLMs: uses · Captioning and enrichment depend on LLM backends
- Embeddings → Embeddings backends: uses · Selects embedding provider
- Pipeline → Storage: writes · Persists indexed content and assets
- Storage → Vector store: uses · Writes vectors through a pluggable store
- Storage → Query engine: prepares · Builds artifacts later consumed by retrieval
- Query engine → Query processing: preprocesses · Normalizes incoming queries
- Query engine → Hybrid search: retrieves · Runs the main retrieval flow
- Hybrid search → Fusion: combines · Merges dense and sparse candidates
- Hybrid search → Reranker: refines · Optionally reranks top results
- Hybrid search → Rerankers: selects · Chooses reranking backend
- Fusion → Response: supplies · Passes ranked chunks to answer assembly
- Response → Citations: adds · Attaches source references
- Response → Multimodal: merges · Combines textual and image context
- Response → LLMs: uses · Generates final answers with model backends
- MCP server → Protocol: speaks · Implements MCP protocol boundary
- MCP server → Ask hub: exposes · Publishes knowledge query tool
- MCP server → List collections: exposes · Publishes collection listing tool
- MCP server → Doc summary: exposes · Publishes document summary tool
- Ask hub → Query engine: calls · Tool routes into retrieval workflow
- List collections → Storage: reads · Lists indexed collections
- Doc summary → Response: uses · Summarizes document content via response assembly
- Tracing → Dashboard: feeds · Dashboard reads trace data
- Trace service → Tracing: queries · Dashboard service accesses trace records
- Evaluation → Tracing: consumes · Evaluation can reuse trace and run data
- Settings → Pipeline: configures · Controls ingestion behavior
- Settings → Query engine: configures · Controls retrieval behavior
- Settings → MCP server: configures · Controls server/runtime options
- Settings → Dashboard: configures · Controls dashboard behavior
- Prompts → Transforms: templates · Supplies refinement and caption prompts
- Prompts → Reranker: templates · Supplies reranking prompt templates