# akshitkhare07/multimodal_rag architecture

> Architecture diagram of the GitHub repository akshitkhare07/multimodal_rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akshitkhare07/multimodal_rag
- Repository: https://github.com/akshitkhare07/multimodal_rag

## Overview

##### What this project is (type + purpose)
This is a **single-process Python Streamlit application** implementing **multimodal RAG**:
- **Inputs**: documents (PDF/TXT/CSV/DOCX/XLSX/JSON) and **videos** (MP4/MOV/…).
- **Processing**: extract text (and for video: audio → transcription via **Whisper**).
- **Indexing**: chunk text + create embeddings; store in **FAISS** vector index.
- **Retrieval + generation**: semantic search over FAISS; pass retrieved context to **Groq LLM** for answers.
- **Persistence**: “session management” to save/load vector stores (likely local disk).

So architecturally it’s a **monolithic interactive app** with clear internal modules and a couple of external dependencies (Groq, Whisper/FFmpeg).

##### What the file tree tells you to model (modules → components)
From `/src`:
- `app.py` — **presentation + orchestration** (Streamlit UI, user flows, session state).
- `data_loader.py` — **document ingestion** (parsing/extraction per file type).
- `video_loader.py` — **video ingestion** (audio extraction + Whisper transcription).
- `ChunkAndEmbed.py` — **text chunking + embedding** pipeline (core indexing transform).
- `vectorStore.py` — **FAISS index lifecycle** (create/update/query/save/load).
- `search.py` — **RAG query path** (retrieval + Groq prompt/LLM call + formatting/sources).

Other notable runtime/config:
- `.streamlit/config.toml` — Streamlit config (UI/server behavior).
- `.streamlit/secrets.toml` (described in README) — **GROQ_API_KEY**.
- `data/` — sample inputs (test PDF, test MP4).
- `DEPLOYMENT_NOTES.md`, `STREAMLIT_CLOUD_SETUP.md` — likely indicate deployment model is Streamlit Cloud (still single service).

##### Ingestion Layer
- Document Loader (`data_loader.py`)
- Video Loader (`video_loader.py`)
##### Indexing Pipeline
- Chunk & Embed (`ChunkAndEmbed.py`)
##### Vector Store Service
- FAISS Store Manager (`vectorStore.py`)
##### RAG Query Service
- Retrieval + Groq LLM (`search.py`)

**External connectors**
- Groq LLM API
- Whisper transcription engine (and FFmpeg if applicable)
- Filesystem persistence

##### `app.py` calls `video_loader.py`
3. `video_loader.py` extracts audio (optionally via FFmpeg) → runs Whisper → transcript (+ timestamps if available)
##### Transcript goes to `ChunkAndEmbed.py` → embeddings
5. `vectorStore.py` upserts into FAISS with metadata `source_type=video`, possibly time ranges

##### Sequence 4 (optional but important): Save/Load session
- `app.py` → `vectorStore.py.save()` writes FAISS index + metadata to disk
- `app.py` → `vectorStore.py.load()` restores index into session

##### Main components to include (and how to label them)
Use these **component blocks** (separate as much as possible):

##### Ingestion
- **Document Parser (`src/data_loader.py`)**
- **Video Processor (`src/video_loader.py`)**
- Audio extraction
- Whisper transcription

##### Indexing
- **Chunker (`src/ChunkAndEmbed.py`)**
- If it’s a hosted embedding API, show it as external; if local, show it as internal library.

##### Storage / Retrieval
- **FAISS Vector Index (`src/vectorStore.py`)**
- **Local Persistence**
- Saved FAISS files + metadata (session save/load)

##### Generation
- **RAG Orchestrator (`src/search.py`)**
- **Groq LLM API** (external)

- Upload → `app.py`: “file bytes + filename + user action”
- `app.py` → `data_loader.py`: “document file → extracted text + metadata”
- `app.py` → `video_loader.py`: “video file → transcript (+ timestamps)”
- Loader → `ChunkAndEmbed.py`: “raw text”
- `ChunkAndEmbed.py` → `vectorStore.py`: “(chunk_id, embedding[], chunk_text, metadata)”
- `search.py` → `vectorStore.py`: “query embedding → topK chunks”
- `search.py` → Groq: “prompt(context + question)”
- Groq → `search.py`: “answer text”
- `vectorStore.py` ↔ Disk: “save/load index + metadata”
- `app.py` ↔ `search.py`: “question + filters → answer + sources”

##### Architectural patterns worth calling out
- **Pipeline pattern** for ingestion/indexing: Load → Normalize text → Chunk → Embed → Upsert.
- **RAG pattern** for query: Embed(query) → Retrieve → Compose prompt → Generate.
- **Session-scoped state**: vector store is “temporary during the session” but can be persisted via save/load.

##### What “very detailed” means for this repo (don’t overthink—split components)
To align with “separating into as many components as possible,” break out these subcomponents even if they live in one file:
- Document loaders by format (PDF vs DOCX vs CSV)
- Video: audio extraction vs transcription
- Chunking vs embedding
- VectorStore: index build/upsert vs query vs persistence
- Search: retrieval vs prompt construction vs LLM call vs citation formatting

That level of detail will produce the most accurate architecture representation for a principal-engineer audience and makes bottlenecks and boundaries immediately visible.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% =========================================================
    %% Level 1 — Context (Environment)
    %% =========================================================
    subgraph "Level 1: Context — Multimodal RAG App in its environment"
        U["User (Browser)"]:::external

        subgraph "Streamlit Service (Single-process Python App)"
            APP["Streamlit Server (UI + Orchestration)"]:::internal
        end

        GROQ["Groq LLM API"]:::external
        FS[("Local Filesystem / Streamlit Cloud Storage")]:::store
        WHISPER["Whisper Runtime (local)"]:::runtime
        FFMPEG["FFmpeg (audio extraction)"]:::runtime
        EMBR["Embedding Model Runtime"]:::runtime
    end

    U -->|"HTTP:ui_interactions"| APP
    APP -->|"HTTPS:prompt+context (GROQ_API_KEY)"| GROQ
    GROQ -->|"answer_text"| APP
    APP -->|"read/write:uploads+saved_indexes"| FS
    APP -->|"transcribe_audio"| WHISPER
    APP -->|"extract_audio"| FFMPEG
    APP -->|"compute_embeddings"| EMBR

    %% =========================================================
    %% Level 2 — Container (Module boundaries within single service)
    %% =========================================================
    subgraph "Level 2: Containers — Internal modules (still one deployable)"
        direction TB

        subgraph "Presentation / Control"
            AAPP["src/app.py\nStreamlit UI + Orchestration\n(Session state: index ref, filters, sources)"]:::internal
            CFG[".streamlit/config.toml\nStreamlit runtime config"]:::config
        end

        subgraph "Ingestion Layer"
            DLOAD["src/data_loader.py\nDocument Parser (PDF/TXT/CSV/DOCX/XLSX/JSON)"]:::internal
            VLOAD["src/video_loader.py\nVideo Processor (audio->Whisper transcript)"]:::internal
        end

        subgraph "Indexing Pipeline"
            CE["src/ChunkAndEmbed.py\nChunking + Embedding Pipeline"]:::internal
        end

        subgraph "Storage / Retrieval"
            VS["src/vectorStore.py\nFAISS Vector Store Manager\n(upsert/query/save/load)"]:::internal
            DISK[("FAISS index files + metadata\n(local persistence)")]:::store
        end

        subgraph "Generation / RAG"
            SVC["src/search.py\nRAG Query Service\n(retrieve->prompt->Groq->citations)"]:::internal
            GROQ2["Groq LLM API"]:::external
        end

        subgraph "Docs / Samples / Dependencies"
            DEP1["DEPLOYMENT_NOTES.md"]:::doc
            DEP2["STREAMLIT_CLOUD_SETUP.md"]:::doc
            DATA["data/\nSample inputs"]:::store
            PDF["data/multimodal_rag_test.pdf"]:::store
            MP4["data/test.mp4"]:::store
            REQ["requirements.txt"]:::doc
        end
    end

    %% Container relationships (vertical, explicit payloads)
    AAPP -->|"doc_file->extract_text+metadata"| DLOAD
    AAPP -->|"video_file->transcript(+timestamps)"| VLOAD
    DLOAD -->|"raw_text"| CE
    VLOAD -->|"transcript_text"| CE
    CE -->|"chunk_id+embedding[]+chunk_text+metadata"| VS

    AAPP -->|"question+filters"| SVC
    SVC -->|"query->topK_request"| VS
    VS -->|"topK_chunks+metadata"| SVC
    SVC -->|"prompt(context+question)"| GROQ2
    GROQ2 -->|"answer_text"| SVC
    SVC -->|"answer+sources"| AAPP

    VS -.->|"save/load:index+metadata"| DISK
    AAPP -.->|"session_save/load triggers"| VS

    AAPP -->|"reads_samples (optional)"| DATA
    DATA -->|"contains"| PDF
    DATA -->|"contains"| MP4

    %% =========================================================
    %% Level 3 — Component detail (inside major modules)
    %% =========================================================
    subgraph "Level 3: Components — Key internal responsibilities"
        direction TB

        subgraph "Document Parser (data_loader.py)"
            direction TB
            DP["Dispatcher: detect file type"]:::internal
            PDFX["PDF extractor"]:::internal
            DOCXX["DOCX extractor"]:::internal
            CSVX["CSV extractor"]:::internal
            XLSXX["XLSX extractor"]:::internal
            JSONX["JSON extractor"]:::internal
            TXTX["TXT extractor"]:::internal
        end

        subgraph "Video Processor (video_loader.py)"
            direction TB
            VUP["Video upload handler"]:::internal
            AUD["Audio extraction (FFmpeg)"]:::runtime
            TR["Whisper transcription"]:::runtime
            TS["Transcript (+timestamps) formatter"]:::internal
        end

        subgraph "Chunk + Embed (ChunkAndEmbed.py)"
            direction TB
            NORM["Text normalize / cleanup"]:::internal
            CHUNK["Chunker (windowing/overlap)"]:::internal
            EM["Embedder (calls embedding runtime)"]:::runtime
        end

        subgraph "Vector Store (vectorStore.py)"
            direction TB
            IDX["FAISS index lifecycle\n(create/load/build)"]:::internal
            UPS["Upsert vectors + metadata"]:::internal
            QRY["Similarity search (topK)"]:::internal
            PERS["Persistence (save/load files)"]:::internal
        end

        subgraph "RAG Query (search.py)"
            direction TB
            QEMB["Query embedding"]:::runtime
            RET["Retriever (FAISS topK)"]:::internal
            PROMPT["Prompt builder\n(context + question)"]:::internal
            LLM["Groq call (chat/completions)"]:::external
            CITE["Citations / sources formatter"]:::internal
        end

        subgraph "Session Scope (app.py)"
            direction TB
            SS["Streamlit session state\n(index_ref, filters, sources registry)"]:::internal
            FLOWS["UI flows:\nUpload Doc, Upload Video, Search, Save/Load"]:::internal
        end
    end

    %% Wiring between components (kept vertical)
    SS -->|"user_upload_doc"| DP
    DP -->|"route_pdf"| PDFX
    DP -->|"route_docx"| DOCXX
    DP -->|"route_csv"| CSVX
    DP -->|"route_xlsx"| XLSXX
    DP -->|"route_json"| JSONX
    DP -->|"route_txt"| TXTX

    PDFX -->|"extracted_text"| NORM
    DOCXX -->|"extracted_text"| NORM
    CSVX -->|"extracted_text"| NORM
    XLSXX -->|"extracted_text"| NORM
    JSONX -->|"extracted_text"| NORM
    TXTX -->|"extracted_text"| NORM

    SS -->|"user_upload_video"| VUP
    VUP -->|"extract_audio"| AUD
    AUD -->|"audio_stream"| TR
    TR -->|"raw_transcript"| TS
    TS -->|"transcript_text"| NORM

    NORM -->|"clean_text"| CHUNK
    CHUNK -->|"chunks[]"| EM
    EM -->|"embeddings[]"| UPS
    IDX -->|"provides_index"| UPS
    UPS -->|"updates_index"| IDX

    SS -->|"user_question+filters"| QEMB
    QEMB -->|"query_embedding"| QRY
    QRY -->|"topK_chunks"| RET
    RET -->|"context_chunks"| PROMPT
    PROMPT -->|"prompt"| LLM
    LLM -->|"answer"| CITE
    CITE -->|"answer+sources"| SS

    PERS -.->|"save/load"| DISK
    IDX -.->|"load_index"| PERS

    %% =========================================================
    %% Click Events (from component mapping)
    %% =========================================================
    click AAPP "https://github.com/akshitkhare07/multimodal_rag/blob/main/src/app.py"
    click DLOAD "https://github.com/akshitkhare07/multimodal_rag/blob/main/src/data_loader.py"
    click VLOAD "https://github.com/akshitkhare07/multimodal_rag/blob/main/src/video_loader.py"
    click CE "https://github.com/akshitkhare07/multimodal_rag/blob/main/src/ChunkAndEmbed.py"
    click VS "https://github.com/akshitkhare07/multimodal_rag/blob/main/src/vectorStore.py"
    click SVC "https://github.com/akshitkhare07/multimodal_rag/blob/main/src/search.py"
    click CFG "https://github.com/akshitkhare07/multimodal_rag/blob/main/.streamlit/config.toml"
    click DEP1 "https://github.com/akshitkhare07/multimodal_rag/blob/main/DEPLOYMENT_NOTES.md"
    click DEP2 "https://github.com/akshitkhare07/multimodal_rag/blob/main/STREAMLIT_CLOUD_SETUP.md"
    click DATA "https://github.com/akshitkhare07/multimodal_rag/tree/main/data/"
    click PDF "https://github.com/akshitkhare07/multimodal_rag/blob/main/data/multimodal_rag_test.pdf"
    click MP4 "https://github.com/akshitkhare07/multimodal_rag/blob/main/data/test.mp4"
    click REQ "https://github.com/akshitkhare07/multimodal_rag/blob/main/requirements.txt"

    %% =========================================================
    %% Styles (color is critical)
    %% =========================================================
    classDef internal fill:#1f77b4,stroke:#0b2d4d,color:#ffffff,stroke-width:1.5px
    classDef external fill:#f59e0b,stroke:#7a4a00,color:#111827,stroke-width:1.5px
    classDef store fill:#22c55e,stroke:#14532d,color:#052e16,stroke-width:1.5px
    classDef runtime fill:#a855f7,stroke:#3b0764,color:#ffffff,stroke-width:1.5px
    classDef config fill:#0ea5e9,stroke:#083344,color:#ffffff,stroke-width:1.5px
    classDef doc fill:#94a3b8,stroke:#334155,color:#0f172a,stroke-width:1.2px
```
