# jiinnjoseph/multi-modal-rag architecture

> Architecture diagram of the GitHub repository jiinnjoseph/multi-modal-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jiinnjoseph/multi-modal-rag
- Repository: https://github.com/jiinnjoseph/multi-modal-rag

## Overview

1. **Overall shape**
   - This is a single-user, Streamlit-driven multimodal RAG application.
   - The runtime is a Python app that accepts uploads, extracts meaning from media, stores embeddings in Qdrant, then retrieves context for LLM answer generation.

2. **UI and orchestration boundary**
   - `app.py` is the top-level orchestrator and user-facing entry point.
   - It likely owns the end-to-end workflow: file upload, ingestion trigger, indexing, question submission, retrieval, and answer rendering.
   - `.streamlit/config.toml` suggests Streamlit-specific UI/runtime configuration.

3. **Upload and file-handling layer**
   - `Utils/upload_utils.py` is the first internal boundary after the UI.
   - This layer likely normalizes uploaded files, validates supported formats, and prepares temporary/local paths for downstream ingestion.

4. **Ingestion subsystem**
   - `Ingestion/ingestion.py` is the multimodal extraction pipeline.
   - It handles different content types: documents, images, audio, and video.
   - The extraction output is not just raw text; it includes transcripts, OCR text, image descriptions, and sampled video-frame content.

5. **Modal-specific processing**
   - Documents are parsed into text chunks.
   - Audio and video are transcribed and likely summarized or described using external model services.
   - Images and video frames go through visual understanding and OCR-style extraction.
   - `yolov8n.pt` indicates object-detection support for video/image frame analysis, so video processing is not purely textual.

6. **Embedding subsystem**
   - `Embeddings/embeddings.py` converts extracted content into vectors.
   - The repo README indicates two embedding families: BGE-style text embeddings and CLIP-style image embeddings.
   - This creates a clear architectural split between text-space retrieval and visual-space retrieval.

7. **Indexing boundary**
   - `Retrieval/indexing.py` is responsible for turning extracted chunks and embeddings into Qdrant records.
   - Payloads likely carry provenance metadata such as source file, chunk/frame identifiers, and modality information.
   - The system stores content into two distinct collections: `rag_text` and `rag_image`.

8. **Vector store and persistence**
   - `Retrieval/qdrant_client.py` is the client abstraction for Qdrant connectivity and collection management.
   - The checked-in `qdrant_storage/` directory shows local persistent Qdrant state, including WAL, shard config, and segment files.
   - That means the repo is set up to work with a local on-disk Qdrant instance in addition to the remote/Qdrant Cloud option described in the README.

9. **Retrieval subsystem**
   - `Retrieval/retrieval.py` performs similarity search over the indexed vectors.
   - Its job is to fetch the most relevant text chunks and/or image-related matches for a user query.
   - The architecture implies hybrid multimodal retrieval rather than a single shared vector space.

10. **Generation subsystem**
   - `Generation/generation.py` produces the final answer from retrieved context.
   - The README says answers are grounded in retrieved material only, so the generator is downstream of retrieval and should not operate as a free-form search engine.
   - `Generation/test.py` suggests there is at least some local validation or prompt/output testing around generation behavior.

11. **External services and model dependencies**
   - The app depends on external LLM/vision services, specifically Groq and Google Gemini per the README.
   - `requirements.txt` likely pulls in the document parsers, OCR/image libs, embedding models, Streamlit, and Qdrant client.
   - Tesseract is an additional optional/local dependency for OCR quality.

12. **End-to-end data flow**
   - Upload in UI → normalize file → ingest/extract modality-specific content → chunk/describe/transcribe → embed text or image content → write to Qdrant collections → query retrieval on user question → feed retrieved context into LLM generation → display grounded answer in Streamlit.
   - The key architectural boundary is between extraction/embedding and retrieval/generation: the first half builds the knowledge base, the second half answers questions from it.

13. **Important architectural distinction for diagrams**
   - Draw two storage lanes: one for text (`rag_text`) and one for images/visual content (`rag_image`).
   - Draw Qdrant as the durable retrieval backend, with the local `qdrant_storage/` directory representing persisted state.
   - Show `app.py` as the control plane, not the processing engine: it coordinates the pipeline, while the subsystem modules perform the work.

Main entry points: [`app.py`](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/app.py) (App), [`Retrieval/retrieval.py`](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/retrieval.py) (Retriever), [`Retrieval/indexing.py`](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/indexing.py) (Indexing), [`Ingestion/ingestion.py`](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Ingestion/ingestion.py) (Ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["UI"]
  node_app["App<br/>streamlit orchestrator<br/>[app.py]"]
  node_streamlit_cfg["Streamlit cfg<br/>ui config<br/>[config.toml]"]
end

subgraph group_ingest["Ingestion"]
  node_upload_utils["Upload utils<br/>file handling<br/>[upload_utils.py]"]
  node_ingestion["Ingestion<br/>multimodal extract<br/>[ingestion.py]"]
end

subgraph group_retrieval["Retrieval"]
  node_embeddings["Embeddings<br/>vectorization<br/>[embeddings.py]"]
  node_indexing["Indexing<br/>qdrant writer<br/>[indexing.py]"]
  node_qdrant_client["Qdrant client<br/>store adapter<br/>[qdrant_client.py]"]
  node_retrieval_core["Retriever<br/>similarity search<br/>[retrieval.py]"]
end

subgraph group_generation["Generation"]
  node_gen_core["Generator<br/>grounded answerer<br/>[generation.py]"]
  node_gen_test["Gen test<br/>validation<br/>[test.py]"]
end

subgraph group_storage["Storage"]
  node_rag_text[("rag_text<br/>text collection")]
  node_rag_image[("rag_image<br/>image collection")]
  node_qdrant_store[("Qdrant store<br/>local persistence")]
end

node_yolo_model["YOLOv8n<br/>vision model<br/>[yolov8n.pt]"]
node_requirements["Dependencies<br/>python deps<br/>[requirements.txt]"]
node_readme["README<br/>project docs<br/>[README.md]"]

node_app -->|"uses"| node_streamlit_cfg
node_app -->|"uploads"| node_upload_utils
node_app -->|"ingests"| node_ingestion
node_app -->|"queries"| node_retrieval_core
node_app -->|"answers"| node_gen_core
node_upload_utils -->|"prepares"| node_ingestion
node_ingestion -->|"extracts"| node_embeddings
node_ingestion -->|"analyzes"| node_yolo_model
node_embeddings -->|"vectors"| node_indexing
node_indexing -->|"writes via"| node_qdrant_client
node_qdrant_client -->|"persists to"| node_qdrant_store
node_indexing -->|"stores"| node_rag_text
node_indexing -->|"stores"| node_rag_image
node_retrieval_core -->|"reads via"| node_qdrant_client
node_retrieval_core -->|"searches"| node_rag_text
node_retrieval_core -->|"searches"| node_rag_image
node_retrieval_core -->|"supplies context"| node_gen_core
node_gen_core -.->|"guided by"| node_readme
node_gen_test -.->|"checks"| node_gen_core
node_requirements -.->|"supports"| node_app

click node_app "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/app.py"
click node_streamlit_cfg "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/.streamlit/config.toml"
click node_upload_utils "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Utils/upload_utils.py"
click node_ingestion "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Ingestion/ingestion.py"
click node_embeddings "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Embeddings/embeddings.py"
click node_indexing "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/indexing.py"
click node_qdrant_client "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/qdrant_client.py"
click node_retrieval_core "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/retrieval.py"
click node_gen_core "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Generation/generation.py"
click node_gen_test "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Generation/test.py"
click node_rag_text "https://github.com/jiinnjoseph/multi-modal-rag/tree/main/qdrant_storage/collections/rag_text"
click node_rag_image "https://github.com/jiinnjoseph/multi-modal-rag/tree/main/qdrant_storage/collections/rag_image"
click node_qdrant_store "https://github.com/jiinnjoseph/multi-modal-rag/tree/main/qdrant_storage"
click node_yolo_model "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/yolov8n.pt"
click node_requirements "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/requirements.txt"
click node_readme "https://github.com/jiinnjoseph/multi-modal-rag/blob/main/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_streamlit_cfg toneBlue
class node_upload_utils,node_ingestion toneAmber
class node_embeddings,node_indexing,node_qdrant_client,node_retrieval_core toneMint
class node_gen_core,node_gen_test toneRose
class node_rag_text,node_rag_image,node_qdrant_store toneIndigo
class node_yolo_model,node_requirements,node_readme toneNeutral
```

## Components

### UI

Streamlit entrypoint and user workflow control

- [App](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/app.py): streamlit orchestrator · Top-level UI and pipeline coordinator · `app.py`
- [Streamlit cfg](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/.streamlit/config.toml): ui config · Streamlit runtime and presentation settings · `.streamlit/config.toml`

### Ingestion

File normalization and multimodal extraction

- [Upload utils](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Utils/upload_utils.py): file handling · Uploaded file validation and normalization · `Utils/upload_utils.py`
- [Ingestion](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Ingestion/ingestion.py): multimodal extract · Parses documents, images, audio, and video into usable content · `Ingestion/ingestion.py`

### Retrieval

Embedding, indexing, search, and persistence

- [Embeddings](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Embeddings/embeddings.py): vectorization · Builds text and image embeddings for downstream search · `Embeddings/embeddings.py`
- [Indexing](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/indexing.py): qdrant writer · Packages chunks and vectors into Qdrant records · `Retrieval/indexing.py`
- [Qdrant client](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/qdrant_client.py): store adapter · Connection and collection management for Qdrant · `Retrieval/qdrant_client.py`
- [Retriever](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Retrieval/retrieval.py): similarity search · Queries the vector store for relevant text and visual matches · `Retrieval/retrieval.py`

### Generation

Answer synthesis from retrieved context

- [Generator](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Generation/generation.py): grounded answerer · Produces final answers from retrieved context · `Generation/generation.py`
- [Gen test](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/Generation/test.py): validation · Local checks around generation behavior · `Generation/test.py`

### Storage

Local durable Qdrant state

- [rag_text](https://github.com/jiinnjoseph/multi-modal-rag/tree/main/qdrant_storage/collections/rag_text): text collection · Persistent text-embedding collection · `qdrant_storage/collections/rag_text`
- [rag_image](https://github.com/jiinnjoseph/multi-modal-rag/tree/main/qdrant_storage/collections/rag_image): image collection · Persistent visual-embedding collection · `qdrant_storage/collections/rag_image`
- [Qdrant store](https://github.com/jiinnjoseph/multi-modal-rag/tree/main/qdrant_storage): local persistence · On-disk Qdrant state including WAL, shards, and segments · `qdrant_storage`

### Other components

- [YOLOv8n](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/yolov8n.pt): vision model · Object detection support for visual/video frame analysis · `yolov8n.pt`
- [Dependencies](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/requirements.txt): python deps · Python package dependencies for UI, OCR, embeddings, and vector search · `requirements.txt`
- [README](https://github.com/jiinnjoseph/multi-modal-rag/blob/main/README.md): project docs · Describes the multimodal RAG workflow and runtime setup · `README.md`

## Connections

- App → Streamlit cfg: uses · UI runtime configuration
- App → Upload utils: uploads · Normalizes user-provided files
- App → Ingestion: ingests · Triggers multimodal extraction
- App → Retriever: queries · Runs question-time retrieval
- App → Generator: answers · Displays grounded responses
- Upload utils → Ingestion: prepares · Hands off validated local file paths
- Ingestion → Embeddings: extracts · Produces text and visual content for vectorization
- Ingestion → YOLOv8n: analyzes · Supports object detection on images and video frames
- Embeddings → Indexing: vectors · Provides embeddings for persistence
- Indexing → Qdrant client: writes via · Uses the Qdrant adapter for collection operations
- Qdrant client → Qdrant store: persists to · Stores and reads local Qdrant state
- Indexing → rag_text: stores · Writes text chunks and metadata
- Indexing → rag_image: stores · Writes image and visual-frame records
- Retriever → Qdrant client: reads via · Searches collections through the Qdrant adapter
- Retriever → rag_text: searches · Retrieves text-space matches
- Retriever → rag_image: searches · Retrieves image-space matches
- Retriever → Generator: supplies context · Passes ranked evidence into generation
- Generator → README: guided by · Grounded-answer behavior described in project docs
- Gen test → Generator: checks · Validates generation behavior
- Dependencies → App: supports · Declares runtime dependencies
