# utkarshk0509/multimodal-rag architecture

> Architecture diagram of the GitHub repository utkarshk0509/multimodal-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/utkarshk0509/multimodal-rag
- Repository: https://github.com/utkarshk0509/multimodal-rag

## Overview

This is a Streamlit multimodal document RAG application. Its primary workflow accepts a PDF, extracts native text or uses Gemini vision for scanned pages, chunks and indexes content in Chroma, then accepts text or voice questions and returns Gemini answers grounded in retrieved passages. Separate scripts support satellite-document parsing, chart description, index initialization, and command-line retrieval. The README provides no additional capability details; relationships shown come from the supplied source excerpts.

Main entry points: [`app.py`](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py) (Streamlit app), [`retrieval/generate_rag_response.py`](https://github.com/utkarshk0509/multimodal-rag/blob/main/retrieval/generate_rag_response.py) (CLI RAG response), [`vector_store/vector_store_init.py`](https://github.com/utkarshk0509/multimodal-rag/blob/main/vector_store/vector_store_init.py) (Index initializer), [`data_ingestion/vision_extractor.py`](https://github.com/utkarshk0509/multimodal-rag/blob/main/data_ingestion/vision_extractor.py) (Chart description)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Document Ingestion"]
  node_app["Streamlit app<br/>[app.py]"]
  node_pdf_extract["PDF extraction<br/>[app.py]"]
  node_vision_ocr["Vision OCR<br/>[app.py]"]
  node_chunking["Text chunking<br/>[app.py]"]
end

subgraph group_index["Index and Retrieval"]
  node_chroma[("Chroma store")]
  node_store_init["Index initializer"]
  node_cli_query["Filtered retrieval<br/>[query_rag.py]"]
end

subgraph group_answer["Question Answering"]
  node_voice_text["Voice and text input<br/>[app.py]"]
  node_retriever["Context retrieval<br/>[app.py]"]
  node_prompt["RAG prompt<br/>[app.py]"]
  node_response["Answer generation<br/>[app.py]"]
  node_cli_answer["CLI RAG response"]
end

subgraph group_tools["Supporting Pipelines"]
  node_advanced_parser["Satellite PDF parser<br/>[advanced_parser.py]"]
  node_pdf_preview["PDF preview utility<br/>[pdf_parser.py]"]
  node_vision_describer["Chart description"]
end

node_user(("Document user"))
node_gemini["Google Gemini"]
node_mic["Microphone transcription"]

node_user -->|"uploads PDF"| node_app
node_app -->|"processes PDF"| node_pdf_extract
node_pdf_extract -->|"OCR fallback"| node_vision_ocr
node_vision_ocr -->|"generates OCR"| node_gemini
node_app -->|"chunks pages"| node_chunking
node_chunking -->|"embeds documents"| node_gemini
node_chunking -->|"persists chunks"| node_chroma
node_user -->|"asks question"| node_voice_text
node_voice_text -->|"transcribes audio"| node_mic
node_voice_text -->|"submits query"| node_retriever
node_retriever -->|"retrieves context"| node_chroma
node_retriever -->|"formats context"| node_prompt
node_prompt -->|"supplies prompt"| node_response
node_response -->|"generates answer"| node_gemini
node_response -->|"displays answer"| node_user
node_advanced_parser -->|"writes images"| node_vision_describer
node_vision_describer -->|"writes descriptions"| node_store_init
node_store_init -->|"indexes documents"| node_chroma
node_cli_query -->|"retrieves profiles"| node_chroma
node_cli_answer -->|"retrieves context"| node_chroma
node_cli_answer -->|"generates answer"| node_gemini

click node_app "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_pdf_extract "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_vision_ocr "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_chunking "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_voice_text "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_retriever "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_prompt "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_response "https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py"
click node_advanced_parser "https://github.com/utkarshk0509/multimodal-rag/blob/main/data_ingestion/advanced_parser.py"
click node_pdf_preview "https://github.com/utkarshk0509/multimodal-rag/blob/main/data_ingestion/pdf_parser.py"
click node_vision_describer "https://github.com/utkarshk0509/multimodal-rag/blob/main/data_ingestion/vision_extractor.py"
click node_store_init "https://github.com/utkarshk0509/multimodal-rag/blob/main/vector_store/vector_store_init.py"
click node_cli_query "https://github.com/utkarshk0509/multimodal-rag/blob/main/retrieval/query_rag.py"
click node_cli_answer "https://github.com/utkarshk0509/multimodal-rag/blob/main/retrieval/generate_rag_response.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_pdf_extract,node_vision_ocr,node_chunking,node_user toneBlue
class node_chroma,node_store_init,node_cli_query toneAmber
class node_voice_text,node_retriever,node_prompt,node_response,node_cli_answer toneMint
class node_advanced_parser,node_pdf_preview,node_vision_describer toneRose
class node_gemini,node_mic toneIndigo
```

## Components

### Document Ingestion

- [Streamlit app](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`
- [PDF extraction](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`
- [Vision OCR](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`
- [Text chunking](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`

### Index and Retrieval

- **Chroma store**
- [Index initializer](https://github.com/utkarshk0509/multimodal-rag/blob/main/vector_store/vector_store_init.py): `vector_store/vector_store_init.py`
- [Filtered retrieval](https://github.com/utkarshk0509/multimodal-rag/blob/main/retrieval/query_rag.py): `retrieval/query_rag.py`

### Question Answering

- [Voice and text input](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`
- [Context retrieval](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`
- [RAG prompt](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`
- [Answer generation](https://github.com/utkarshk0509/multimodal-rag/blob/main/app.py): `app.py`
- [CLI RAG response](https://github.com/utkarshk0509/multimodal-rag/blob/main/retrieval/generate_rag_response.py): `retrieval/generate_rag_response.py`

### Supporting Pipelines

- [Satellite PDF parser](https://github.com/utkarshk0509/multimodal-rag/blob/main/data_ingestion/advanced_parser.py): `data_ingestion/advanced_parser.py`
- [PDF preview utility](https://github.com/utkarshk0509/multimodal-rag/blob/main/data_ingestion/pdf_parser.py): `data_ingestion/pdf_parser.py`
- [Chart description](https://github.com/utkarshk0509/multimodal-rag/blob/main/data_ingestion/vision_extractor.py): `data_ingestion/vision_extractor.py`

### Other components

- **Document user**
- **Google Gemini**
- **Microphone transcription**

## Connections

- Document user → Streamlit app: uploads PDF (evidence: `app.py`)
- Streamlit app → PDF extraction: processes PDF (evidence: `app.py`)
- PDF extraction → Vision OCR: OCR fallback (evidence: `app.py`)
- Vision OCR → Google Gemini: generates OCR (evidence: `app.py`)
- Streamlit app → Text chunking: chunks pages (evidence: `app.py`)
- Text chunking → Google Gemini: embeds documents (evidence: `app.py`)
- Text chunking → Chroma store: persists chunks (evidence: `app.py`)
- Document user → Voice and text input: asks question (evidence: `app.py`)
- Voice and text input → Microphone transcription: transcribes audio (evidence: `app.py`)
- Voice and text input → Context retrieval: submits query (evidence: `app.py`)
- Context retrieval → Chroma store: retrieves context (evidence: `app.py`)
- Context retrieval → RAG prompt: formats context (evidence: `app.py`)
- RAG prompt → Answer generation: supplies prompt (evidence: `app.py`)
- Answer generation → Google Gemini: generates answer (evidence: `app.py`)
- Answer generation → Document user: displays answer (evidence: `app.py`)
- Satellite PDF parser → Chart description: writes images (evidence: `data_ingestion/advanced_parser.py`)
- Chart description → Index initializer: writes descriptions (evidence: `data_ingestion/vision_extractor.py`)
- Index initializer → Chroma store: indexes documents (evidence: `vector_store/vector_store_init.py`)
- Filtered retrieval → Chroma store: retrieves profiles (evidence: `retrieval/query_rag.py`)
- CLI RAG response → Chroma store: retrieves context (evidence: `retrieval/generate_rag_response.py`)
- CLI RAG response → Google Gemini: generates answer (evidence: `retrieval/generate_rag_response.py`)
