How muhammedsaidali/agentic-multimodal-hybrid-rag-system works
This is a single-process, local-first Streamlit application that implements an agentic multimodal Hybrid RAG system. It ingests user-provided PDFs + images (and optionally web content), performs task-aware orchestration, uses explicit evidence selection, and then runs hybrid retrieval (vector + BM25) to ground an LLM response. Models are served locally via Ollama.
• Interactive UI → orchestration/agent → retrieval pipelines → LLM reasoning → grounded response
• The evidence-selection gate (core design decision)
• The multimodal path (images → vision-to-text → treated as textual evidence)
• The fact that vector store is in-memory and ephemeral
• External dependencies: Ollama, ChromaDB, BM25, BeautifulSoup/web, LangChain
File structure analysis (map code to components)
Given the tree:
• app.py → Streamlit UI layer (upload PDFs/images, preview, checkboxes for selection, chat UX)
• advanced_engine.py → Core engine:
• task classification / routing
• document parsing & chunking
• embedding + vector indexing (Chroma in-memory)
• BM25 indexing
• hybrid retrieval & reranking/merging
• image vision-to-text conversion (via Ollama vision model)
• prompt assembly with citations/evidence
• LLM calls via Ollama (reasoning model + embedding model)
• optional web scraping (BeautifulSoup)
• requirements.txt → confirms stack (Streamlit, LangChain, Chroma, BM25 libs, bs4, etc.)
• start_ollama.bat → indicates the local runtime dependency (Ollama process)
Streamlit UI (app.py)
• Upload PDFs
• Upload images
• Evidence preview
• Evidence selection checkboxes
• Chat input/output
Task Classifier / Router
• determines intent: fact / analysis / summary / compare / explore
Evidence Selection Gate
• uses *only* user-selected PDFs/images
• fallback behavior: if nothing selected → trigger retrieval over all ingested content (per README)
PDF Ingestion Pipeline
• parse PDF → text
• chunk / normalize
Image Understanding Pipeline
• image → vision-to-text description (Ollama vision model, e.g., “moondream”)
• store description as textual artifact (no image embeddings by design)
Web Ingestion (optional)
• URL fetch → HTML parse (BeautifulSoup) → text chunks
Embedding Service (Ollama embedding model)
• e.g., nomic-embed-text
Vector Store (ChromaDB in-memory)
• embeddings index, ephemeral lifecycle
BM25 Index
• keyword retrieval index (in-memory)
Hybrid Retrieval Combiner
• run vector search + BM25
• merge results (and optionally dedupe / rank)
• output: “selected evidence snippets”
Prompt Builder / Grounding Formatter
• injects retrieved chunks + image descriptions
• includes “explicit evidence selection” context and constraints
LLM Reasoner (Ollama LLM, e.g., llama3)
• produces final response grounded in evidence
Grounded Response
• returned to Streamlit chat UI
• (optionally) includes citations/snippet references
User → Streamlit UI: question + (optional) evidence selections
2. Streamlit UI → Agent/Engine: query + selected_pdf_ids + selected_image_ids + settings
Agent → Evidence Selection Gate
• If user selected evidence: constrain corpus to selected items
• Else: allow retrieval over all ingested items (fallback)
Agent → Hybrid Retrieval
• BM25 search over text chunks
• Embedding service call → vector search in Chroma
Ingestion/indexing flow (happens on upload / session start)
• Streamlit UI → PDF Ingestion → Chunking → (Embeddings → Chroma) + (BM25 index)
• Streamlit UI → Image Pipeline → Vision-to-text (Ollama vision model) → store text → BM25 (+ optionally embed as text if implemented)
• in-memory indices cleared when app stops
Architectural patterns/design principles to highlight visually
• Explicit evidence selection as a first-class control point (gate before retrieval/prompting)
• Hybrid RAG (two retrieval strategies in parallel: vector + BM25)
• Multimodal via text normalization (images become textual descriptions; no image embeddings)
• Local model runtime (Ollama as an external system dependency)
• Ephemeral state (in-memory Chroma/BM25; session-scoped, not durable)
User ↔ Streamlit App ↔ Ollama; optional web sources.
Keep retrieval internals abstract (just “Hybrid RAG Engine”).
Break Streamlit vs Engine, and then show subcomponents inside engine:
• Task classifier
• Evidence selection gate
• PDF ingestion
• Image V2T
• BM25 index
• Chroma in-memory vector store
• Hybrid combiner
• Prompt builder
• Ollama embedding model vs Ollama LLM vs Ollama vision model
One for:
• “User selects evidence” path
• “No selection → fallback to hybrid retrieval” path
Components with source references:
• “Streamlit UI (app.py)”
• “Agent + Retrieval Engine (advanced_engine.py)”
• “Ollama Model Config (Modelfile)”
• “Ollama runtime start script (start_ollama.bat)”
The “don’t overthink it” rule applied here (maximize component separation)
• Task classifier (separate)
• Evidence gate (separate)
• PDF vs image pipelines (separate)
• BM25 vs vector retrieval (separate)
• Prompt builder (separate)
• LLM call (separate)
• Embedding call (separate)
• Vision call (separate)
That separation is exactly what communicates the project’s main contribution: controlled multimodal hybrid RAG with agentic routing.