# aranya-chatterjee/multimodal_rag architecture

> Architecture diagram of the GitHub repository aranya-chatterjee/multimodal_rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aranya-chatterjee/multimodal_rag
- Repository: https://github.com/aranya-chatterjee/multimodal_rag

## Overview

1. **Overall shape**
   - This is a single-process **Streamlit** application that acts as a multimodal RAG front end.
   - The main flow is: **upload file → extract text/transcript → chunk and embed → store in FAISS → search/retrieve → send context to Groq LLM → show answer and sources**.

2. **UI / orchestration layer**
   - `src/app.py` is the top-level application entry point and likely owns the Streamlit sidebar/main-page interaction.
   - It orchestrates file upload, source type selection, session actions like saving/loading, and user queries.
   - The UI is the only user-facing boundary; all processing happens in the supporting modules.

3. **Document ingestion subsystem**
   - `src/data_loader.py` handles non-video inputs.
   - Per the README, it supports PDF, TXT, CSV, DOCX, XLSX, and JSON, so this module is the document normalization boundary.
   - Its output is presumably plain text or document chunks suitable for downstream embedding.

4. **Video ingestion subsystem**
   - `src/video_loader.py` handles MP4/MOV/AVI/MKV/WMV inputs.
   - The README says it performs **audio transcription**, so the video path is effectively: video file → extracted audio/transcript → text for RAG.
   - This is the main multimodal boundary in the repo: video content is converted into searchable text before entering the same retrieval pipeline as documents.

5. **Chunking and embedding**
   - `src/ChunkAndEmbed.py` is the text preparation layer.
   - It likely splits loaded text/transcripts into chunks and generates embeddings.
   - This module is the bridge between raw content and retrieval-ready vector representations, so it is central to both document and video ingestion.

6. **Vector storage**
   - `src/vectorStore.py` manages the **FAISS-based vector store** mentioned in the README.
   - This is the persistence/search index layer for embeddings and associated metadata.
   - The repository treats the vector store as session-scoped state with optional save/load behavior, rather than a long-lived external database.

7. **Retrieval and answer generation**
   - `src/search.py` performs semantic search over the vector store and connects retrieval results to the **Groq LLM**.
   - This module is the RAG “reasoning” boundary: it takes a user question, retrieves relevant chunks, and produces an answer grounded in the retrieved context.
   - It also appears to surface source references, which means retrieved chunks are preserved with enough metadata to cite them back in the UI.

8. **Data flow for document questions**
   - For documents, the path is: upload in Streamlit → `data_loader.py` parses content → `ChunkAndEmbed.py` creates chunks and embeddings → `vectorStore.py` stores them → `search.py` retrieves relevant chunks → Groq generates the response.
   - This is a standard text-only RAG flow.

9. **Data flow for video questions**
   - For videos, the path is: upload in Streamlit → `video_loader.py` transcribes audio → the transcript goes through the same chunk/embed/store/search pipeline.
   - The important architectural point is that video is not searched directly; it is normalized into text first, then merged into the same retrieval index as documents.

10. **State and lifecycle**
   - The app appears to keep the vector store in memory during the Streamlit session and optionally persist it to disk via “Save Session.”
   - That makes the system lightweight and self-contained, but it also means retrieval state is application-managed rather than backed by a database service.

11. **External dependencies that matter architecturally**
   - **Groq API** is the external LLM runtime used for generation.
   - **Whisper** is used for video transcription, which is why video ingestion is slower and more compute-heavy than document ingestion.
   - **FAISS** is the local vector search engine, so semantic retrieval is done in-process rather than via a hosted search service.

12. **Configuration and deployment boundaries**
   - `.streamlit/config.toml` and `.streamlit/secrets.toml` are important operational files.
   - The app depends on a secret `GROQ_API_KEY`, so the LLM integration is environment-configured rather than hardcoded.
   - The presence of `DEPLOYMENT_NOTES.md` and `STREAMLIT_CLOUD_SETUP.md` suggests the app is intended to run locally or on Streamlit Cloud with minimal infrastructure.

13. **Repository structure summary**
   - `src/app.py` = orchestration/UI.
   - `src/data_loader.py` + `src/video_loader.py` = ingestion and normalization.
   - `src/ChunkAndEmbed.py` = chunking/embedding transformation.
   - `src/vectorStore.py` = vector index management.
   - `src/search.py` = retrieval + LLM answer synthesis.
   - `data/` contains sample inputs for testing the pipeline end to end.

14. **How to diagram it**
   - Draw a single Streamlit UI at the top, with two ingestion branches: documents and videos.
   - Merge both branches into a shared chunk/embed stage, then into a FAISS vector store.
   - From the query side, show user questions flowing into search/retrieval, then into Groq for response generation, and finally back to the UI with answers plus source references.

Main entry points: [`src/app.py`](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/app.py) (Streamlit app), [`src/search.py`](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/search.py) (RAG answer), [`src/vectorStore.py`](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/vectorStore.py) (FAISS store), [`src/video_loader.py`](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/video_loader.py) (Video loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_ui["UI & Orchestration"]
  node_node_app["Streamlit app<br/>entrypoint<br/>[app.py]"]
  node_node_sidebar["Controls<br/>ui flow<br/>[app.py]"]
  node_node_answer_view["Answer view<br/>ui output<br/>[app.py]"]
  node_node_user(("User<br/>human actor"))
end

subgraph group_group_ingest["Ingestion"]
  node_node_doc_loader["Doc loader<br/>parser<br/>[data_loader.py]"]
  node_node_video_loader["Video loader<br/>transcriber<br/>[video_loader.py]"]
end

subgraph group_group_retrieval["Retrieval & Generation"]
  node_node_chunker["Chunker<br/>text prep<br/>[ChunkAndEmbed.py]"]
  node_node_embedder["Embedder<br/>embeddings<br/>[ChunkAndEmbed.py]"]
  node_node_vector_store[("FAISS store<br/>vector index<br/>[vectorStore.py]")]
  node_node_index_ops["Store ops<br/>persistence<br/>[vectorStore.py]"]
  node_node_search["Search<br/>retriever<br/>[search.py]"]
  node_node_rag{{"RAG answer<br/>llm synthesis<br/>[search.py]"}}
  node_node_sources["Sources<br/>citations<br/>[search.py]"]
end

subgraph group_group_external["External Services"]
  node_node_groq(("Groq API<br/>llm runtime"))
  node_node_whisper(("Whisper<br/>speech-to-text"))
  node_node_faiss[("FAISS<br/>vector engine")]
end

node_node_user -->|"uploads / asks"| node_node_app
node_node_app -->|"routes actions"| node_node_sidebar
node_node_app -->|"document input"| node_node_doc_loader
node_node_app -->|"video input"| node_node_video_loader
node_node_video_loader -->|"transcribes"| node_node_whisper
node_node_doc_loader -->|"raw text"| node_node_chunker
node_node_video_loader -->|"transcript text"| node_node_chunker
node_node_chunker -->|"chunks"| node_node_embedder
node_node_embedder -->|"vectors"| node_node_vector_store
node_node_vector_store -->|"index backend"| node_node_faiss
node_node_app -->|"save / load"| node_node_index_ops
node_node_index_ops -->|"manages"| node_node_vector_store
node_node_user -->|"questions"| node_node_search
node_node_search -->|"retrieve"| node_node_vector_store
node_node_search -->|"context"| node_node_rag
node_node_rag -->|"generates"| node_node_groq
node_node_rag -->|"attaches citations"| node_node_sources
node_node_rag -->|"renders"| node_node_answer_view
node_node_answer_view -->|"response"| node_node_user

click node_node_app "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/app.py"
click node_node_sidebar "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/app.py"
click node_node_answer_view "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/app.py"
click node_node_doc_loader "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/data_loader.py"
click node_node_video_loader "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/video_loader.py"
click node_node_chunker "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/ChunkAndEmbed.py"
click node_node_embedder "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/ChunkAndEmbed.py"
click node_node_vector_store "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/vectorStore.py"
click node_node_index_ops "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/vectorStore.py"
click node_node_search "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/search.py"
click node_node_rag "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/search.py"
click node_node_sources "https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/search.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_app,node_node_sidebar,node_node_answer_view,node_node_user toneBlue
class node_node_doc_loader,node_node_video_loader toneAmber
class node_node_chunker,node_node_embedder,node_node_vector_store,node_node_index_ops,node_node_search,node_node_rag,node_node_sources toneMint
class node_node_groq,node_node_whisper,node_node_faiss toneRose
```

## Components

### UI & Orchestration

Streamlit entry point and session flow

- [Streamlit app](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/app.py): entrypoint · Top-level UI and orchestration for uploads, queries, and session actions · `src/app.py`
- [Controls](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/app.py): ui flow · Source selection, upload, and session management interactions · `src/app.py`
- [Answer view](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/app.py): ui output · Displays responses, sources, and current session state · `src/app.py`
- **User**: human actor · Uploads content and asks questions through the Streamlit interface

### Ingestion

Document and video normalization into text

- [Doc loader](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/data_loader.py): parser · Normalizes PDF, TXT, CSV, DOCX, XLSX, and JSON inputs into text · `src/data_loader.py`
- [Video loader](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/video_loader.py): transcriber · Extracts audio and transcribes video files into text · `src/video_loader.py`

### Retrieval & Generation

Chunking, vector search, and answer synthesis

- [Chunker](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/ChunkAndEmbed.py): text prep · Splits normalized text into retrieval-sized chunks · `src/ChunkAndEmbed.py`
- [Embedder](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/ChunkAndEmbed.py): embeddings · Creates vector representations for chunks and transcripts · `src/ChunkAndEmbed.py`
- [FAISS store](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/vectorStore.py): vector index · In-process vector store for saving and searching embedded chunks · `src/vectorStore.py`
- [Store ops](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/vectorStore.py): persistence · Session save/load and index lifecycle management · `src/vectorStore.py`
- [Search](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/search.py): retriever · Semantic retrieval over the vector store for user questions · `src/search.py`
- [RAG answer](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/search.py): llm synthesis · Combines retrieved context with the question to generate a grounded answer · `src/search.py`
- [Sources](https://github.com/aranya-chatterjee/multimodal_rag/blob/main/src/search.py): citations · Carries chunk metadata used to show references back in the UI · `src/search.py`

### External Services

Third-party runtime dependencies

- **Groq API**: llm runtime · External model provider used for response generation
- **Whisper**: speech-to-text · External transcription runtime used for video audio extraction
- **FAISS**: vector engine · Local nearest-neighbor search engine backing semantic retrieval

## Connections

- User → Streamlit app: uploads / asks · User interacts with the Streamlit app
- Streamlit app → Controls: routes actions · App coordinates upload, source selection, and session controls
- Streamlit app → Doc loader: document input · Document files are normalized into text
- Streamlit app → Video loader: video input · Video files are sent for transcription
- Video loader → Whisper: transcribes · Audio is converted to transcript text
- Doc loader → Chunker: raw text · Parsed documents feed the chunking stage
- Video loader → Chunker: transcript text · Transcripts join the same chunking pipeline
- Chunker → Embedder: chunks · Text chunks are embedded for retrieval
- Embedder → FAISS store: vectors · Embeddings and metadata are stored for later search
- FAISS store → FAISS: index backend · FAISS provides local vector similarity search
- Streamlit app → Store ops: save / load · Session state can be persisted and restored
- Store ops → FAISS store: manages · Persistence layer owns the vector index lifecycle
- User → Search: questions · User queries enter retrieval through the app
- Search → FAISS store: retrieve · Search pulls relevant chunks from the vector index
- Search → RAG answer: context · Retrieved passages are passed into answer synthesis
- RAG answer → Groq API: generates · Groq produces the final grounded response
- RAG answer → Sources: attaches citations · Source metadata is preserved for display
- RAG answer → Answer view: renders · Answer and sources are shown in the UI
- Answer view → User: response · Final output returns to the user
