How cornelliusyw/multimodal-rag-implementation works
Project Type and Purpose
• This is a specialized data‐processing & inference pipeline (an open-source tool) for Multimodal Retrieval‐Augmented Generation (RAG).
• It ingests PDFs and audio, converts them into embeddings, stores them in a vector database, and uses a multimodal LLM (Qwen-VL) to answer user queries by retrieving relevant image/text/audio chunks.
File Structure Analysis
• Top level:
• RAG_Multimodal.ipynb – Jupyter notebook orchestrating the full pipeline.
• README.md – Project description, prerequisites, pipeline steps.
• dataset/ – Raw assets:
◦ recipe_book.pdf – PDF source for image extraction
◦ Quick & Easy Recipes With Gordon Ramsay.mp3 – Audio source for transcription
• No separate “src” or “scripts” folder; the notebook likely includes all orchestration, util functions and model calls inline.
Key Pipeline Components
a. Data Ingestion & Preprocessing
• PDF‐to‐image converter (pdf2image, Pillow)
• Audio loader & chunker (librosa)
b. Embedding Generation
• CLIP model (from transformers) for images
• Whisper (from transformers) for audio‐to‐text transcription
• SentenceTransformer for text embedding of the transcriptions
c. Vector Storage & Retrieval
• ChromaDB collections for image and text embeddings
• Similarity search API to fetch top-k relevant items
d. Multimodal Query & Generation
• Qwen-VL receives: original user query + retrieved image embeddings + retrieved text embeddings + (optionally) audio context
• Generates multimodal‐aware text response
e. Presentation Layer
• Notebook cells display retrieved images and final text outputs
Relationships and Data Flow
• PDF → Images → Image embeddings → ChromaDB (image collection)
• Audio → Whisper transcription → Text chunks → Text embeddings → ChromaDB (text collection)
• User query →
• Compute query embedding (SentenceTransformer)
• Query ChromaDB image & text collections → Top-k similar items
• Gather retrieved images + text → Feed into Qwen-VL along with query → Generated answer
• Images & answer back to user
Architectural Patterns & Principles
• Modular pipeline: each stage (ingest, embed, store, retrieve, generate) is a self-contained step
• Separation of concerns:
• Preprocessing
• Embedding vs. storage vs. retrieval
• Inference (Qwen-VL)
• External service integration: ChromaDB as an external vector DB, transformer models as “stateless” processing units
• Dataflow architecture: directed acyclic graph of transformations