# cornelliusyw/multimodal-rag-implementation architecture

> Architecture diagram of the GitHub repository cornelliusyw/multimodal-rag-implementation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cornelliusyw/multimodal-rag-implementation
- Repository: https://github.com/cornelliusyw/multimodal-rag-implementation

## Overview

##### Project Type and Purpose
- This is a specialized **data‐processing & inference pipeline** (an open-source tool) for **Multimodal Retrieval‐Augmented Generation (RAG)**.
- It ingests PDFs and audio, converts them into embeddings, stores them in a vector database, and uses a multimodal LLM (Qwen-VL) to answer user queries by retrieving relevant image/text/audio chunks.

##### File Structure Analysis
- Top level:
- RAG_Multimodal.ipynb – Jupyter notebook orchestrating the full pipeline.
- README.md – Project description, prerequisites, pipeline steps.
- dataset/ – Raw assets:
◦ recipe_book.pdf – PDF source for image extraction
◦ Quick & Easy Recipes With Gordon Ramsay.mp3 – Audio source for transcription
- No separate “src” or “scripts” folder; the notebook likely includes all orchestration, util functions and model calls inline.

##### Key Pipeline Components
a. **Data Ingestion & Preprocessing**
- PDF‐to‐image converter (pdf2image, Pillow)
- Audio loader & chunker (librosa)
b. **Embedding Generation**
- CLIP model (from transformers) for images
- Whisper (from transformers) for audio‐to‐text transcription
- SentenceTransformer for text embedding of the transcriptions
c. **Vector Storage & Retrieval**
- ChromaDB collections for image and text embeddings
- Similarity search API to fetch top-k relevant items
d. **Multimodal Query & Generation**
- Qwen-VL receives: original user query + retrieved image embeddings + retrieved text embeddings + (optionally) audio context
- Generates multimodal‐aware text response
e. **Presentation Layer**
- Notebook cells display retrieved images and final text outputs

##### Relationships and Data Flow
- PDF → Images → Image embeddings → ChromaDB (image collection)
- Audio → Whisper transcription → Text chunks → Text embeddings → ChromaDB (text collection)
- User query →
- Compute query embedding (SentenceTransformer)
- Query ChromaDB image & text collections → Top-k similar items
- Gather retrieved images + text → Feed into Qwen-VL along with query → Generated answer
- Images & answer back to user

##### Architectural Patterns & Principles
- **Modular pipeline**: each stage (ingest, embed, store, retrieve, generate) is a self-contained step
- **Separation of concerns**:
- Preprocessing
- Embedding vs. storage vs. retrieval
- Inference (Qwen-VL)
- **External service integration**: ChromaDB as an external vector DB, transformer models as “stateless” processing units
- **Dataflow architecture**: directed acyclic graph of transformations

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraphs for pipeline stages
    subgraph "Preprocessing"
        direction TB
        RawPDF["PDF Source\n(recipe_book.pdf)"]:::source
        PDFIngestion["PDF Ingestion & Preprocessing"]:::source
        RawAudio["Audio Source\n(Quick & Easy Recipes With Gordon Ramsay.mp3)"]:::source
        AudioIngestion["Audio Ingestion & Chunking"]:::source

        RawPDF -->|Extract images| PDFIngestion
        RawAudio -->|Load & chunk| AudioIngestion
    end

    subgraph "Embedding & Storage"
        direction TB
        ImageEmbedding["Image Embedding Generation\n(CLIP ViT-L/14)"]:::process
        AudioTranscription["Audio Transcription\n(Whisper Base)"]:::process
        TextEmbedding["Text Embedding Generation\n(SentenceTransformer)"]:::process
        ImageStore["ChromaDB Image Store"]:::storage
        TextStore["ChromaDB Text Store"]:::storage

        PDFIngestion -->|Images| ImageEmbedding -->|Image embeddings| ImageStore
        AudioIngestion -->|Audio chunks| AudioTranscription -->|Transcripts| TextEmbedding -->|Text embeddings| TextStore
    end

    subgraph "Retrieval"
        direction TB
        Query["User Query"]:::source
        QueryEmbedding["Query Embedding\n(SentenceTransformer)"]:::process
        RetrievalAPI["Similarity Search / Retrieval API"]:::process
        Retrieved["Retrieved Items\n(Top-k images & texts)"]:::process

        Query -->|Text| QueryEmbedding -->|Query embedding| RetrievalAPI
        ImageStore -->|Query| RetrievalAPI
        TextStore -->|Query| RetrievalAPI
        RetrievalAPI -->|Items| Retrieved
    end

    subgraph "Generation"
        direction TB
        QwenVL["Multimodal Query & Generation\n(Qwen-VL)"]:::process

        Retrieved -->|Context| QwenVL -->|Generated Response| NotebookUI
    end

    subgraph "Presentation"
        direction TB
        NotebookUI["Presentation Layer\n(Notebook UI)"]:::presentation
        README["Project Overview & Instructions\n(README.md)"]:::presentation

        NotebookUI -->|View| README
    end

    %% Click Events
    click PDFIngestion "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click AudioIngestion "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click ImageEmbedding "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click AudioTranscription "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click TextEmbedding "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click ImageStore "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click TextStore "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click RetrievalAPI "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click QwenVL "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click NotebookUI "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/RAG_Multimodal.ipynb"
    click RawPDF "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/dataset/recipe_book.pdf"
    click RawAudio "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/dataset/Quick & Easy Recipes With Gordon Ramsay.mp3"
    click README "https://github.com/cornelliusyw/multimodal-rag-implementation/blob/main/README.md"

    %% Styles
    classDef source fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef process fill:#90EE90,stroke:#000,stroke-width:1px
    classDef storage fill:#FFA500,stroke:#000,stroke-width:1px
    classDef presentation fill:#D8BFD8,stroke:#000,stroke-width:1px
```
