# vijay-punyamurthy/multimodal-rag-pipeline architecture

> Architecture diagram of the GitHub repository vijay-punyamurthy/multimodal-rag-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vijay-punyamurthy/multimodal-rag-pipeline
- Repository: https://github.com/vijay-punyamurthy/multimodal-rag-pipeline

## Overview

This is a local, multimodal PDF question-answering system: users ingest a document, then retrieve text and relevant diagrams or tables to generate a grounded answer with a local vision-language model. The CLI and Flask UI both invoke the same pipeline. Ingestion separates OCR text, native table extraction, and cropped visuals; querying combines dense and BM25 text recall, reranking, page expansion, and cross-modal image filtering. The graph follows the documented and excerpted calls; external model services are shown separately.

Main entry points: [`pipeline.py`](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/pipeline.py) (Pipeline), [`retriever.py`](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/retriever.py) (Retriever), [`extractor.py`](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py) (OCR + Tables), [`config.py`](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/config.py) (Configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_cli["CLI<br/>[main.py]"]
  node_web["Web UI<br/>[app.py]"]
  node_pipeline["Pipeline<br/>[pipeline.py]"]
end

subgraph group_ingestion["PDF Ingestion"]
  node_extract["OCR + Tables<br/>[extractor.py]"]
  node_visuals["Visual Crops<br/>[extractor.py]"]
  node_text_items["Text Chunks<br/>[extractor.py]"]
  node_image_items["Image Chunks<br/>[extractor.py]"]
  node_text_embed["Text Embedder<br/>[embeddings.py]"]
  node_clip_embed["CLIP Embedder<br/>[embeddings.py]"]
end

subgraph group_retrieval["Hybrid Retrieval"]
  node_store[("Vector Store<br/>[vector_store.py]")]
  node_bm25[("BM25 Index<br/>[bm25_index.py]")]
  node_retriever["Retriever<br/>[retriever.py]"]
  node_rerank["Cross-Encoder Rerank<br/>[retriever.py]"]
  node_image_filter["Image Filter<br/>[retriever.py]"]
end

subgraph group_generation["Answer Generation"]
  node_generator["Answer Generator<br/>[generator.py]"]
  node_ollama["Ollama VLM"]
  node_answer["Grounded Answer<br/>[generator.py]"]
end

node_user(("User"))
node_pdf["PDF Document"]
node_config["Configuration<br/>[config.py]"]

node_user -->|"runs command"| node_cli
node_user -->|"uses UI"| node_web
node_pdf -->|"supplies input"| node_cli
node_cli -->|"calls ingest/query"| node_pipeline
node_web -->|"calls ingest/query"| node_pipeline
node_pipeline -->|"extracts PDF"| node_extract
node_extract -->|"returns OCR/table text"| node_text_items
node_visuals -->|"produces image chunks"| node_image_items
node_extract -->|"detects visual content"| node_visuals
node_pipeline -->|"embeds text chunks"| node_text_embed
node_pipeline -->|"embeds images"| node_clip_embed
node_pipeline -->|"adds embeddings"| node_store
node_pipeline -->|"builds index"| node_bm25
node_pipeline -->|"retrieves context"| node_retriever
node_retriever -->|"embeds query"| node_text_embed
node_retriever -->|"embeds image query"| node_clip_embed
node_retriever -->|"queries vectors"| node_store
node_retriever -->|"searches keywords"| node_bm25
node_retriever -->|"scores candidates"| node_rerank
node_retriever -->|"filters image hits"| node_image_filter
node_pipeline -->|"generates answer"| node_generator
node_generator -->|"sends prompt and images"| node_ollama
node_ollama -->|"returns response"| node_answer
node_pipeline -.->|"uses settings indirectly"| node_config
node_extract -->|"reads extraction settings"| node_config
node_retriever -->|"reads retrieval settings"| node_config
node_generator -->|"reads model setting"| node_config

click node_cli "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/main.py"
click node_web "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/app.py"
click node_pipeline "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/pipeline.py"
click node_config "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/config.py"
click node_extract "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py"
click node_visuals "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py"
click node_text_items "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py"
click node_image_items "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py"
click node_text_embed "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/embeddings.py"
click node_clip_embed "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/embeddings.py"
click node_store "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/vector_store.py"
click node_bm25 "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/bm25_index.py"
click node_retriever "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/retriever.py"
click node_rerank "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/retriever.py"
click node_image_filter "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/retriever.py"
click node_generator "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/generator.py"
click node_answer "https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/generator.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_web,node_pipeline,node_user toneBlue
class node_extract,node_visuals,node_text_items,node_image_items,node_text_embed,node_clip_embed toneAmber
class node_store,node_bm25,node_retriever,node_rerank,node_image_filter toneMint
class node_generator,node_ollama,node_answer toneRose
class node_pdf toneIndigo
class node_config toneTeal
```

## Components

### User Interfaces

- [CLI](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/main.py): `main.py`
- [Web UI](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/app.py): `app.py`
- [Pipeline](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/pipeline.py): `pipeline.py`

### PDF Ingestion

- [OCR + Tables](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py): `extractor.py`
- [Visual Crops](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py): `extractor.py`
- [Text Chunks](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py): `extractor.py`
- [Image Chunks](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/extractor.py): `extractor.py`
- [Text Embedder](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/embeddings.py): `embeddings.py`
- [CLIP Embedder](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/embeddings.py): `embeddings.py`

### Hybrid Retrieval

- [Vector Store](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/vector_store.py): `vector_store.py`
- [BM25 Index](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/bm25_index.py): `bm25_index.py`
- [Retriever](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/retriever.py): `retriever.py`
- [Cross-Encoder Rerank](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/retriever.py): `retriever.py`
- [Image Filter](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/retriever.py): `retriever.py`

### Answer Generation

- [Answer Generator](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/generator.py): `generator.py`
- **Ollama VLM**
- [Grounded Answer](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/generator.py): `generator.py`

### Other components

- **User**
- **PDF Document**
- [Configuration](https://github.com/vijay-punyamurthy/multimodal-rag-pipeline/blob/main/config.py): `config.py`

## Connections

- User → CLI: runs command (evidence: `main.py`)
- User → Web UI: uses UI (evidence: `app.py`)
- PDF Document → CLI: supplies input (evidence: `main.py`)
- CLI → Pipeline: calls ingest/query (evidence: `main.py`)
- Web UI → Pipeline: calls ingest/query (evidence: `app.py`)
- Pipeline → OCR + Tables: extracts PDF (evidence: `pipeline.py`)
- OCR + Tables → Text Chunks: returns OCR/table text (evidence: `extractor.py`)
- Visual Crops → Image Chunks: produces image chunks (evidence: `extractor.py`)
- OCR + Tables → Visual Crops: detects visual content (evidence: `extractor.py`)
- Pipeline → Text Embedder: embeds text chunks (evidence: `pipeline.py`)
- Pipeline → CLIP Embedder: embeds images (evidence: `pipeline.py`)
- Pipeline → Vector Store: adds embeddings (evidence: `pipeline.py`)
- Pipeline → BM25 Index: builds index (evidence: `pipeline.py`)
- Pipeline → Retriever: retrieves context (evidence: `pipeline.py`)
- Retriever → Text Embedder: embeds query (evidence: `retriever.py`)
- Retriever → CLIP Embedder: embeds image query (evidence: `retriever.py`)
- Retriever → Vector Store: queries vectors (evidence: `retriever.py`)
- Retriever → BM25 Index: searches keywords (evidence: `retriever.py`)
- Retriever → Cross-Encoder Rerank: scores candidates (evidence: `retriever.py`)
- Retriever → Image Filter: filters image hits (evidence: `retriever.py`)
- Pipeline → Answer Generator: generates answer (evidence: `pipeline.py`)
- Answer Generator → Ollama VLM: sends prompt and images (evidence: `generator.py`)
- Ollama VLM → Grounded Answer: returns response (evidence: `generator.py`)
- Pipeline → Configuration: uses settings indirectly
- OCR + Tables → Configuration: reads extraction settings (evidence: `extractor.py`)
- Retriever → Configuration: reads retrieval settings (evidence: `retriever.py`)
- Answer Generator → Configuration: reads model setting (evidence: `generator.py`)
