# muhammedsaidali/agentic-multimodal-hybrid-rag-system architecture

> Architecture diagram of the GitHub repository muhammedsaidali/agentic-multimodal-hybrid-rag-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system
- Repository: https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system

## Overview

This is a **single-process, local-first Streamlit application** that implements an **agentic multimodal Hybrid RAG system**. It ingests **user-provided PDFs + images** (and optionally **web content**), performs **task-aware orchestration**, uses **explicit evidence selection**, and then runs **hybrid retrieval (vector + BM25)** to ground an **LLM response**. Models are served locally via **Ollama**.

- **Interactive UI → orchestration/agent → retrieval pipelines → LLM reasoning → grounded response**
- The **evidence-selection gate** (core design decision)
- The **multimodal path** (images → vision-to-text → treated as textual evidence)
- The fact that **vector store is in-memory and ephemeral**
- External dependencies: **Ollama**, **ChromaDB**, **BM25**, **BeautifulSoup/web**, **LangChain**

##### File structure analysis (map code to components)
Given the tree:
- `app.py` → **Streamlit UI layer** (upload PDFs/images, preview, checkboxes for selection, chat UX)
- `advanced_engine.py` → **Core engine**:
- task classification / routing
- document parsing & chunking
- embedding + vector indexing (Chroma in-memory)
- BM25 indexing
- hybrid retrieval & reranking/merging
- image vision-to-text conversion (via Ollama vision model)
- prompt assembly with citations/evidence
- LLM calls via Ollama (reasoning model + embedding model)
- optional web scraping (BeautifulSoup)
- `requirements.txt` → confirms stack (Streamlit, LangChain, Chroma, BM25 libs, bs4, etc.)
- `start_ollama.bat` → indicates the local runtime dependency (Ollama process)

##### Streamlit UI (app.py)
- Upload PDFs
- Upload images
- Evidence preview
- Evidence selection checkboxes
- Chat input/output

##### Task Classifier / Router
- determines intent: fact / analysis / summary / compare / explore
##### Evidence Selection Gate
- uses *only* user-selected PDFs/images
- fallback behavior: if nothing selected → trigger retrieval over all ingested content (per README)

##### PDF Ingestion Pipeline
- parse PDF → text
- chunk / normalize
##### Image Understanding Pipeline
- image → vision-to-text description (Ollama vision model, e.g., “moondream”)
- store description as textual artifact (no image embeddings by design)
##### Web Ingestion (optional)
- URL fetch → HTML parse (BeautifulSoup) → text chunks

##### Embedding Service (Ollama embedding model)
- e.g., `nomic-embed-text`
##### Vector Store (ChromaDB in-memory)
- embeddings index, ephemeral lifecycle
##### BM25 Index
- keyword retrieval index (in-memory)
##### Hybrid Retrieval Combiner
- run vector search + BM25
- merge results (and optionally dedupe / rank)
- output: “selected evidence snippets”

##### Prompt Builder / Grounding Formatter
- injects retrieved chunks + image descriptions
- includes “explicit evidence selection” context and constraints
##### LLM Reasoner (Ollama LLM, e.g., llama3)
- produces final response grounded in evidence

##### Grounded Response
- returned to Streamlit chat UI
- (optionally) includes citations/snippet references

##### User → Streamlit UI: question + (optional) evidence selections
2. **Streamlit UI → Agent/Engine**: `query + selected_pdf_ids + selected_image_ids + settings`
##### Agent → Evidence Selection Gate
- If user selected evidence: constrain corpus to selected items
- Else: allow retrieval over all ingested items (fallback)
##### Agent → Hybrid Retrieval
- **BM25 search** over text chunks
- **Embedding service** call → vector search in **Chroma**

##### Ingestion/indexing flow (happens on upload / session start)
- **Streamlit UI → PDF Ingestion → Chunking → (Embeddings → Chroma) + (BM25 index)**
- **Streamlit UI → Image Pipeline → Vision-to-text (Ollama vision model) → store text → BM25 (+ optionally embed as text if implemented)**
- **in-memory indices cleared when app stops**

##### Architectural patterns/design principles to highlight visually
- **Explicit evidence selection** as a first-class control point (gate before retrieval/prompting)
- **Hybrid RAG** (two retrieval strategies in parallel: vector + BM25)
- **Multimodal via text normalization** (images become textual descriptions; no image embeddings)
- **Local model runtime** (Ollama as an external system dependency)
- **Ephemeral state** (in-memory Chroma/BM25; session-scoped, not durable)

User ↔ Streamlit App ↔ Ollama; optional web sources.
Keep retrieval internals abstract (just “Hybrid RAG Engine”).

Break Streamlit vs Engine, and then show subcomponents inside engine:
- Task classifier
- Evidence selection gate
- PDF ingestion
- Image V2T
- BM25 index
- Chroma in-memory vector store
- Hybrid combiner
- Prompt builder
- Ollama embedding model vs Ollama LLM vs Ollama vision model

One for:
- “User selects evidence” path
- “No selection → fallback to hybrid retrieval” path

Components with source references:
- “Streamlit UI (app.py)”
- “Agent + Retrieval Engine (advanced_engine.py)”
- “Ollama Model Config (Modelfile)”
- “Ollama runtime start script (start_ollama.bat)”

##### The “don’t overthink it” rule applied here (maximize component separation)
- Task classifier (separate)
- Evidence gate (separate)
- PDF vs image pipelines (separate)
- BM25 vs vector retrieval (separate)
- Prompt builder (separate)
- LLM call (separate)
- Embedding call (separate)
- Vision call (separate)

That separation is exactly what communicates the project’s main contribution: **controlled multimodal hybrid RAG with agentic routing**.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ============
    %% Global Actors
    %% ============
    U["User"]:::actor

    %% =========================
    %% Presentation / UI Layer
    %% =========================
    subgraph "Presentation (Local Streamlit App)"
        UI["Streamlit UI\n• Upload PDFs/images\n• Evidence preview + selection\n• Chat I/O"]:::ui
        RESP["Grounded Response\n(with citations/snippets)"]:::output
    end

    %% =========================
    %% Orchestration / Engine
    %% =========================
    subgraph "Application + Domain Logic (Core Engine)"
        ENG["Agent / Orchestration / Core Hybrid RAG Engine"]:::engine

        TC["Task Classifier / Router\n(intent: fact/analysis/summary/compare/explore)"]:::orchestrator
        GATE["Evidence Selection Gate (user-controlled)\n• Use only selected items\n• Fallback: none selected → all ingested"]:::gate

        subgraph "Ingestion / Preprocessing (session-scoped)"
            PDF["PDF Ingestion Pipeline\n(parse → normalize → chunk)"]:::pipeline
            IMG["Image Understanding Pipeline\n(vision-to-text → textual artifact)\n(no image embeddings by design)"]:::pipeline
            WEB["Web Ingestion (optional)\n(fetch → BeautifulSoup parse → chunk)"]:::pipeline
        end

        subgraph "Hybrid RAG Retrieval (parallel)"
            EMB["Embedding Service\n(Ollama embedding model)"]:::externalSvc
            VEC[("Vector Store\nChromaDB (in-memory)\nRAM / session-scoped")]:::ephemeral
            BM25[("BM25 Index\n(in-memory)\nRAM / session-scoped")]:::ephemeral
            HYB["Hybrid Retrieval Combiner\nVector + BM25 merge/dedupe/rerank\n→ top-k evidence snippets"]:::retrieval
        end

        PB["Prompt Builder / Grounding Formatter\n(inject evidence + constraints + citations)"]:::prompt
        LLM["LLM Reasoner\n(Ollama LLM)"]:::externalSvc
    end

    %% =========================
    %% External Dependencies
    %% =========================
    subgraph "External / Local Dependencies"
        OLL["Ollama Local Runtime\n(LLM + Embeddings + Vision models)"]:::external
        VISION["Ollama Vision Model\n(e.g., moondream)"]:::external
        WEBEXT["Web Sources / URLs"]:::external
        NOTE["Note:\nIndices are ephemeral (in-memory) and cleared when app stops"]:::note
    end

    %% =========================
    %% Primary runtime query flow
    %% =========================
    U -->|"asks_question + selects_evidence"| UI
    UI -->|"query + selected_pdf_ids + selected_image_ids + settings"| ENG
    ENG -->|"classify_query"| TC
    TC -->|"route/intention"| GATE
    GATE -->|"constrained_corpus_or_fallback_all"| HYB
    HYB -->|"evidence_snippets + image_description_text"| PB
    PB -->|"grounded_prompt + citations"| LLM
    LLM -->|"answer_text"| RESP
    RESP -->|"render_chat"| UI
    UI -->|"display_answer"| U

    %% =========================
    %% Ingestion/indexing flow (dashed)
    %% =========================
    UI -.->|"upload_pdfs"| PDF
    PDF -.->|"text_chunks"| BM25
    PDF -.->|"text_chunks"| EMB
    EMB -.->|"embeddings"| VEC

    UI -.->|"upload_images"| IMG
    IMG -.->|"image_bytes"| VISION
    VISION -.->|"image_description_text"| IMG
    IMG -.->|"description_chunks"| BM25
    IMG -.->|"description_chunks (as text)"| EMB

    UI -.->|"optional_urls"| WEB
    WEBEXT -.->|"fetch_content"| WEB
    WEB -.->|"text_chunks"| BM25
    WEB -.->|"text_chunks"| EMB

    %% =========================
    %% Retrieval internals (parallel)
    %% =========================
    HYB -->|"bm25_search(query)"| BM25
    HYB -->|"embed(query)"| EMB
    HYB -->|"vector_search(query_embedding)"| VEC
    BM25 -->|"keyword_hits"| HYB
    VEC -->|"semantic_hits"| HYB

    %% =========================
    %% External runtime linkage
    %% =========================
    LLM -->|"runs_on"| OLL
    EMB -->|"runs_on"| OLL
    VISION -->|"runs_on"| OLL

    %% Notes
    NOTE -.->|"applies_to"| VEC
    NOTE -.->|"applies_to"| BM25

    %% =========================
    %% Click Events (from component mapping)
    %% =========================
    click UI "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/app.py"
    click ENG "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click TC "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click GATE "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click PDF "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click IMG "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click WEB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click EMB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click VEC "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click BM25 "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click HYB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click PB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click LLM "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click OLL "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/start_ollama.bat"
    click VISION "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click NOTE "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/requirements.txt"
    click WEBEXT "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click RESP "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"

    %% Additional mapped files
    click OLL "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/start_ollama.bat"
    click PB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click LLM "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click EMB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click VEC "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click BM25 "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click HYB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click TC "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click GATE "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click PDF "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click IMG "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click WEB "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/advanced_engine.py"
    click UI "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/app.py"
    click NOTE "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/requirements.txt"
    click OLL "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/blob/main/start_ollama.bat"
    click OLLCFG "https://github.com/muhammedsaidali/agentic-multimodal-hybrid-rag-system/tree/main/Modelfile"

    %% Define a hidden node for Modelfile (config) to attach click without showing paths
    OLLCFG["Ollama Model Configuration\n(Modelfile)"]:::external
    OLLCFG -.->|"configures_models"| OLL

    %% =========================
    %% Styles
    %% =========================
    classDef actor fill:#0b1320,stroke:#94a3b8,color:#e5e7eb,stroke-width:1.2px
    classDef ui fill:#1d4ed8,stroke:#0b2a6f,color:#ffffff,stroke-width:1.2px
    classDef output fill:#0ea5e9,stroke:#075985,color:#ffffff,stroke-width:1.2px

    classDef engine fill:#6d28d9,stroke:#3b0764,color:#ffffff,stroke-width:1.2px
    classDef orchestrator fill:#7c3aed,stroke:#4c1d95,color:#ffffff,stroke-width:1.2px
    classDef gate fill:#a21caf,stroke:#701a75,color:#ffffff,stroke-width:1.5px

    classDef pipeline fill:#f59e0b,stroke:#92400e,color:#111827,stroke-width:1.2px
    classDef retrieval fill:#22c55e,stroke:#166534,color:#052e16,stroke-width:1.2px
    classDef ephemeral fill:#86efac,stroke:#166534,color:#052e16,stroke-width:1.6px
    classDef prompt fill:#14b8a6,stroke:#115e59,color:#062a27,stroke-width:1.2px

    classDef external fill:#6b7280,stroke:#374151,color:#ffffff,stroke-width:1.2px
    classDef externalSvc fill:#9ca3af,stroke:#4b5563,color:#111827,stroke-width:1.2px
    classDef note fill:#fde68a,stroke:#b45309,color:#111827,stroke-width:1.2px
```
