# dubeamit/multimodal-rag architecture

> Architecture diagram of the GitHub repository dubeamit/multimodal-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dubeamit/multimodal-rag
- Repository: https://github.com/dubeamit/multimodal-rag

## Overview

This is a multimodal RAG application for uploading PDFs and videos, asking grounded questions across a session’s files, and navigating answers back to cited pages or timestamps. The main workflow runs from the browser chat through FastAPI ingestion or querying, then through modality-specific extraction and session-scoped vector retrieval to a local or Gemini model. The interface persists sessions locally and routes citations to its PDF viewer or video player. Some frontend upload and citation-rendering details are only partially sampled; those links follow the README and visible component interfaces.

Main entry points: [`frontend/src/app/page.tsx`](https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/app/page.tsx) (Session and media pane), [`backend/main.py`](https://github.com/dubeamit/multimodal-rag/blob/main/backend/main.py) (FastAPI endpoints), [`backend/vector_store.py`](https://github.com/dubeamit/multimodal-rag/blob/main/backend/vector_store.py) (Chunk indexing and RAG), [`frontend/src/components/ChatInterface.tsx`](https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/components/ChatInterface.tsx) (Chat and citations)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Browser experience"]
  node_page["Session and media pane<br/>[page.tsx]"]
  node_chat["Chat and citations<br/>[ChatInterface.tsx]"]
  node_pdf["PDF viewer<br/>[PdfViewer.tsx]"]
end

subgraph group_api["API orchestration"]
  node_api["FastAPI endpoints<br/>[main.py]"]
end

subgraph group_ingest["Document ingestion"]
  node_processor["PDF and video processing"]
  node_transcriber["Audio transcription runner<br/>[transcribe.sh]"]
end

subgraph group_rag["Retrieval and generation"]
  node_vector["Chunk indexing and RAG<br/>[vector_store.py]"]
  node_chroma[("Session vector collection<br/>[vector_store.py]")]
end

node_user(("User"))
node_local_llm{{"Local OpenAI-compatible model"}}
node_gemini{{"Gemini model"}}

node_user -->|"uses"| node_page
node_page -->|"renders"| node_chat
node_chat -->|"uploads or queries"| node_api
node_api -->|"processes files"| node_processor
node_api -->|"indexes chunks"| node_vector
node_processor -->|"runs transcription"| node_transcriber
node_api -->|"queries RAG"| node_vector
node_vector -->|"embeds, stores, retrieves"| node_chroma
node_vector -->|"requests answer"| node_local_llm
node_vector -.->|"requests answer"| node_gemini
node_api -->|"returns answer and citations"| node_chat
node_chat -->|"dispatches citation"| node_page
node_page -->|"sets cited page"| node_pdf
node_page -->|"seeks video timestamp"| node_page

click node_page "https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/app/page.tsx"
click node_chat "https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/components/ChatInterface.tsx"
click node_pdf "https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/components/PdfViewer.tsx"
click node_api "https://github.com/dubeamit/multimodal-rag/blob/main/backend/main.py"
click node_processor "https://github.com/dubeamit/multimodal-rag/blob/main/backend/multimodal_processor.py"
click node_transcriber "https://github.com/dubeamit/multimodal-rag/blob/main/backend/speech2text/transcribe.sh"
click node_vector "https://github.com/dubeamit/multimodal-rag/blob/main/backend/vector_store.py"
click node_chroma "https://github.com/dubeamit/multimodal-rag/blob/main/backend/vector_store.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_page,node_chat,node_pdf,node_user toneBlue
class node_api toneAmber
class node_processor,node_transcriber toneMint
class node_vector,node_chroma toneRose
class node_local_llm,node_gemini toneIndigo
```

## Components

### Browser experience

- [Session and media pane](https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/app/page.tsx): `frontend/src/app/page.tsx`
- [Chat and citations](https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/components/ChatInterface.tsx): `frontend/src/components/ChatInterface.tsx`
- [PDF viewer](https://github.com/dubeamit/multimodal-rag/blob/main/frontend/src/components/PdfViewer.tsx): `frontend/src/components/PdfViewer.tsx`

### API orchestration

- [FastAPI endpoints](https://github.com/dubeamit/multimodal-rag/blob/main/backend/main.py): `backend/main.py`

### Document ingestion

- [PDF and video processing](https://github.com/dubeamit/multimodal-rag/blob/main/backend/multimodal_processor.py): `backend/multimodal_processor.py`
- [Audio transcription runner](https://github.com/dubeamit/multimodal-rag/blob/main/backend/speech2text/transcribe.sh): `backend/speech2text/transcribe.sh`

### Retrieval and generation

- [Chunk indexing and RAG](https://github.com/dubeamit/multimodal-rag/blob/main/backend/vector_store.py): `backend/vector_store.py`
- [Session vector collection](https://github.com/dubeamit/multimodal-rag/blob/main/backend/vector_store.py): `backend/vector_store.py`

### Other components

- **User**
- **Local OpenAI-compatible model**
- **Gemini model**

## Connections

- User → Session and media pane: uses
- Session and media pane → Chat and citations: renders
- Chat and citations → FastAPI endpoints: uploads or queries
- FastAPI endpoints → PDF and video processing: processes files
- FastAPI endpoints → Chunk indexing and RAG: indexes chunks
- PDF and video processing → Audio transcription runner: runs transcription
- FastAPI endpoints → Chunk indexing and RAG: queries RAG
- Chunk indexing and RAG → Session vector collection: embeds, stores, retrieves
- Chunk indexing and RAG → Local OpenAI-compatible model: requests answer
- Chunk indexing and RAG → Gemini model: requests answer
- FastAPI endpoints → Chat and citations: returns answer and citations
- Chat and citations → Session and media pane: dispatches citation
- Session and media pane → PDF viewer: sets cited page
- Session and media pane → Session and media pane: seeks video timestamp