How olvenint/multi-format-rag works
This is a multi-format document RAG system. Users upload Word, PDF, TXT, or Excel files, which are parsed into chunks and indexed; they then ask questions through the web UI or REST API and receive document-grounded, optionally streamed answers. The main stages are ingestion, hybrid retrieval and ranking, answer generation, and session memory. README capabilities include image descriptions, incremental updates, Redis caching, and multi-session isolation. Some subsystem links are documented in the README; source excerpts are partial, so uncertain calls are omitted.
Main entry points: knowledge_base.py Knowledge base ingestion · qa_service.py Hybrid retrieval · loader_factory.py Loader factory · app_api.py REST and SSE API
Components 22
Service interface
- Chat and upload UI
static/index.html - REST and SSE API
app_api.py
Document ingestion
- Loader factory
loader_factory.py - Word parser
loaders/docx_loader.py - PDF parser
loaders/pdf_loader.py - TXT parser
loaders/txt_loader.py - Excel parser
loaders/excel_loader.py - Loader contract
loaders/base_loader.py - Knowledge base ingestion
knowledge_base.py
Retrieval and ranking
- Term extraction
term_extractor.py - Query rewriting
query_rewriter.py - Hybrid retrieval
qa_service.py - BM25 index
bm25_index.py - Rerank service
reranker.py
Answer and memory
- Answer orchestration
qa_service.py - Conversation memory
memory.py
Persistence and integrations
- Chroma vector store
knowledge_base.py - Redis cache and sessions
redis_cache.py - DeepSeek model
- Vision model
- Runtime configuration
config_data.py
Other components
- User
Connections 32
- User → Chat and upload UI: uses
README.md - Chat and upload UI → REST and SSE API: calls
README.md - User → REST and SSE API: calls
README.md - REST and SSE API → Loader factory: dispatches uploads
README.md - REST and SSE API → Knowledge base ingestion: ingests documents
README.md - REST and SSE API → Answer orchestration: submits questions
README.md - Loader factory → Word parser: creates
loader_factory.py - Loader factory → PDF parser: creates
loader_factory.py - Loader factory → TXT parser: creates
loader_factory.py - Loader factory → Excel parser: creates
loader_factory.py - Word parser → Loader contract: implements
loaders/docx_loader.py - PDF parser → Loader contract: implements
loaders/pdf_loader.py - TXT parser → Loader contract: implements
loaders/txt_loader.py - Excel parser → Loader contract: implements
loaders/excel_loader.py - Knowledge base ingestion → Chroma vector store: writes chunks
README.md - Knowledge base ingestion → Vision model: describes images
README.md - Knowledge base ingestion → Term extraction: updates terms
term_extractor.py - Knowledge base ingestion → Redis cache and sessions: clears caches
knowledge_base.py - Answer orchestration → Conversation memory: loads and saves
qa_service.py - Answer orchestration → Hybrid retrieval: requests passages
qa_service.py - Hybrid retrieval → Query rewriting: rewrites query
qa_service.py - Hybrid retrieval → Chroma vector store: vector searches
README.md - Hybrid retrieval → BM25 index: lexical searches
README.md - Hybrid retrieval → Rerank service: reranks candidates
README.md - Hybrid retrieval → Redis cache and sessions: reads and writes cache
qa_service.py - Answer orchestration → DeepSeek model: generates answer
README.md - Answer orchestration → Redis cache and sessions: checks answer cache
README.md - Term extraction → Query rewriting: supplies vocabulary
query_rewriter.py - Conversation memory → Redis cache and sessions: stores sessions
README.md - Conversation memory → Runtime configuration: reads settings
memory.py - Knowledge base ingestion → Runtime configuration: reads settings
knowledge_base.py - Answer orchestration → Runtime configuration: reads settings
qa_service.py