How haris525/local-multimodal-rag works
Local Multimodal RAG lets signed-in users ingest private documents and ask cited questions over selected files. The main workflow runs from upload through parsing, structured-table extraction, chunking and indexing, then question planning, hybrid retrieval and answer generation; chats, accounts, audit, and optional analysis services support it. The README documents several service-level connections not visible in sampled implementation excerpts, so those edges cite the README. Unsampled modules are represented only where the README or source index supports their role.
Main entry points: rag/api.py RAG API · app.js Web UI · pipeline.py Document pipeline · ingest/api.py Upload and registry API
Components 23
User access
- Web UI
services/gateway/static/app.js - Accounts and sessions
services/auth/src/auth/api.py
Document ingestion
- Upload and registry API
services/ingest/src/ingest/api.py - Ingestion worker
services/ingest/src/ingest/worker.py - Document pipeline
services/ingest/src/ingest/pipeline.py - Format parsers
services/ingest/src/ingest/parsers - Chunking
services/ingest/src/ingest/chunking.py - Index writer
services/ingest/src/ingest/indexer.py
Question answering
- RAG API
services/rag/src/rag/api.py - Query planning
services/rag/src/rag/planner.py - Hybrid retrieval
services/rag/src/rag/retrieval.py - Deep search agent
services/rag/src/rag/agent.py - Read-all mode
services/rag/src/rag/exhaustive.py - Embedding and reranking
services/models/src/models_svc/encoders.py - Local language model
State and services
- Service contracts
libs/common/src/rag_common/contracts.py - Vector and BM25 index
- Read-only SQL service
services/sql/src/sql_sandbox/api.py - Python sandbox
services/sandbox/src/sandbox/api.py - Chats and memory
services/threads/src/threads/api.py - Audit log
services/audit/src/audit/api.py
Evaluation
- RAG evaluation
eval/src/rag_eval/cli.py
Other components
- User
Connections 23
- User → Web UI: asks through
README.md - Web UI → Accounts and sessions: signs in via
README.md - Web UI → Upload and registry API: uploads documents
README.md - Web UI → RAG API: asks questions
README.md - Upload and registry API → Ingestion worker: queues documents
README.md - Ingestion worker → Document pipeline: runs jobs
services/ingest/src/ingest/worker.py - Document pipeline → Format parsers: parses files
services/ingest/src/ingest/pipeline.py - Document pipeline → Chunking: chunks content
services/ingest/src/ingest/pipeline.py - Document pipeline → Index writer: writes index
services/ingest/src/ingest/pipeline.py - Index writer → Vector and BM25 index: writes vectors
README.md - RAG API → Query planning: routes questions
README.md - Query planning → Hybrid retrieval: selects search
README.md - Hybrid retrieval → Vector and BM25 index: searches index
README.md - Hybrid retrieval → Embedding and reranking: embeds and reranks
README.md - RAG API → Deep search agent: runs deep search
README.md - RAG API → Read-all mode: runs read-all
README.md - RAG API → Local language model: generates answers
README.md - RAG API → Chats and memory: loads and saves chats
README.md - Query planning → Read-only SQL service: queries tables
README.md - Deep search agent → Python sandbox: may run Python
README.md - Ingestion worker → Audit log: records events
README.md - Upload and registry API → Service contracts: uses shared models
- RAG evaluation → RAG API: evaluates answers
README.md