How shreyashdarade/modular-rag-pipeline works
This is a modular, multilingual RAG service with REST/SSE and MCP interfaces, an optional Python SDK, and separate ingestion workers. The main flow is client request → API/application use case → retrieval or ingestion → answer/indexed corpus. Collections, parsers, models, queues, and runtime backends are configurable adapters. The graph includes documented evaluation and scaling capabilities; some subsystem links rely on README documentation or source-index references because their implementations were not excerpted.
Main entry points: container.py Composition root · ingestion/service.py Ingestion pipeline · pipeline.py Retrieval pipeline · application/service.py Use-case service
Components 22
Interfaces
- REST and SSE API
src/api/server.py - Python SDK
ai_rag_info/client.py - MCP tools
src/mcp_server/server.py - CLI and evaluation
src/cli/app.py - RAG evaluation
src/evaluation/session.py
Application
- Use-case service
src/application/service.py - Settings and specs
src/core/specs.py - Composition root
src/core/container.py
Ingestion
- Ingestion pipeline
src/ingestion/service.py - Upload storage
src/ingestion/storage.py - File parsing and OCR
src/parsing/registry.py - Chunking
src/chunking/recursive.py
Retrieval and chat
- Retrieval pipeline
src/retrieval/pipeline.py - Hybrid search
src/retrieval/hybrid.py - Model providers
src/models/registry.py - Model rerankers
src/models/rerankers.py - Answer generation
src/chat/answer.py - Conversation service
src/chat/service.py
Runtime infrastructure
- Ingestion job queues
src/jobs/redis_streams.py - Elasticsearch index
src/indexing/elastic.py - Cache and runtime
src/runtime/cache.py
Other components
- API client
Connections 29
- API client → REST and SSE API: requests
README.md - REST and SSE API → Composition root: builds services
src/api/server.py - REST and SSE API → Use-case service: routes use cases
README.md - Python SDK → Use-case service: uses shared cases
README.md - MCP tools → Composition root: uses services
src/mcp_server/server.py - CLI and evaluation → Composition root: uses configuration
src/cli/app.py - CLI and evaluation → RAG evaluation: runs evaluation
README.md - Composition root → Settings and specs: loads configuration
src/core/container.py - Composition root → Ingestion pipeline: constructs pipeline
src/core/container.py - Composition root → Retrieval pipeline: constructs pipeline
src/core/container.py - Composition root → Conversation service: constructs service
src/core/container.py - Use-case service → Ingestion pipeline: coordinates ingestion
src/application/service.py - Use-case service → Upload storage: stores uploads
src/application/service.py - Ingestion pipeline → File parsing and OCR: selects parsers
src/ingestion/service.py - Ingestion pipeline → Chunking: chunks documents
src/ingestion/service.py - Ingestion pipeline → Model providers: embeds content
src/ingestion/service.py - Ingestion pipeline → Elasticsearch index: writes documents
src/ingestion/service.py - Ingestion job queues → Ingestion pipeline: dispatches jobs
README.md - Retrieval pipeline → Hybrid search: retrieves candidates
src/retrieval/pipeline.py - Retrieval pipeline → Cache and runtime: caches results
src/retrieval/pipeline.py - Hybrid search → Elasticsearch index: searches indices
src/retrieval/hybrid.py - Hybrid search → Model providers: embeds queries
src/retrieval/hybrid.py - Retrieval pipeline → Model rerankers: reranks candidates
README.md - Answer generation → Retrieval pipeline: gets evidence
src/chat/answer.py - Answer generation → Model providers: generates answer
src/chat/answer.py - Conversation service → Answer generation: answers turns
src/chat/service.py - Conversation service → Cache and runtime: stores conversations
README.md - RAG evaluation → Retrieval pipeline: evaluates retrieval
README.md - RAG evaluation → Answer generation: evaluates answers
README.md