How jiinnjoseph/multi-modal-rag works
1. Overall shape
- This is a single-user, Streamlit-driven multimodal RAG application.
- The runtime is a Python app that accepts uploads, extracts meaning from media, stores embeddings in Qdrant, then retrieves context for LLM answer generation.
2. UI and orchestration boundary
- app.py is the top-level orchestrator and user-facing entry point.
- It likely owns the end-to-end workflow: file upload, ingestion trigger, indexing, question submission, retrieval, and answer rendering.
- .streamlit/config.toml suggests Streamlit-specific UI/runtime configuration.
3. Upload and file-handling layer
- Utils/upload_utils.py is the first internal boundary after the UI.
- This layer likely normalizes uploaded files, validates supported formats, and prepares temporary/local paths for downstream ingestion.
4. Ingestion subsystem
- Ingestion/ingestion.py is the multimodal extraction pipeline.
- It handles different content types: documents, images, audio, and video.
- The extraction output is not just raw text; it includes transcripts, OCR text, image descriptions, and sampled video-frame content.
5. Modal-specific processing
- Documents are parsed into text chunks.
- Audio and video are transcribed and likely summarized or described using external model services.
- Images and video frames go through visual understanding and OCR-style extraction.
- yolov8n.pt indicates object-detection support for video/image frame analysis, so video processing is not purely textual.
6. Embedding subsystem
- Embeddings/embeddings.py converts extracted content into vectors.
- The repo README indicates two embedding families: BGE-style text embeddings and CLIP-style image embeddings.
- This creates a clear architectural split between text-space retrieval and visual-space retrieval.
7. Indexing boundary
- Retrieval/indexing.py is responsible for turning extracted chunks and embeddings into Qdrant records.
- Payloads likely carry provenance metadata such as source file, chunk/frame identifiers, and modality information.
- The system stores content into two distinct collections: rag_text and rag_image.
8. Vector store and persistence
- Retrieval/qdrant_client.py is the client abstraction for Qdrant connectivity and collection management.
- The checked-in qdrant_storage/ directory shows local persistent Qdrant state, including WAL, shard config, and segment files.
- That means the repo is set up to work with a local on-disk Qdrant instance in addition to the remote/Qdrant Cloud option described in the README.
9. Retrieval subsystem
- Retrieval/retrieval.py performs similarity search over the indexed vectors.
- Its job is to fetch the most relevant text chunks and/or image-related matches for a user query.
- The architecture implies hybrid multimodal retrieval rather than a single shared vector space.
10. Generation subsystem
- Generation/generation.py produces the final answer from retrieved context.
- The README says answers are grounded in retrieved material only, so the generator is downstream of retrieval and should not operate as a free-form search engine.
- Generation/test.py suggests there is at least some local validation or prompt/output testing around generation behavior.
11. External services and model dependencies
- The app depends on external LLM/vision services, specifically Groq and Google Gemini per the README.
- requirements.txt likely pulls in the document parsers, OCR/image libs, embedding models, Streamlit, and Qdrant client.
- Tesseract is an additional optional/local dependency for OCR quality.
12. End-to-end data flow
- Upload in UI → normalize file → ingest/extract modality-specific content → chunk/describe/transcribe → embed text or image content → write to Qdrant collections → query retrieval on user question → feed retrieved context into LLM generation → display grounded answer in Streamlit.
- The key architectural boundary is between extraction/embedding and retrieval/generation: the first half builds the knowledge base, the second half answers questions from it.
13. Important architectural distinction for diagrams
- Draw two storage lanes: one for text (rag_text) and one for images/visual content (rag_image).
- Draw Qdrant as the durable retrieval backend, with the local qdrant_storage/ directory representing persisted state.
- Show app.py as the control plane, not the processing engine: it coordinates the pipeline, while the subsystem modules perform the work.
Main entry points: app.py App · retrieval.py Retriever · indexing.py Indexing · ingestion.py Ingestion
Components 16
UI
Streamlit entrypoint and user workflow control
- App · Top-level UI and pipeline coordinator
app.py - Streamlit cfg · Streamlit runtime and presentation settings
.streamlit/config.toml
Ingestion
File normalization and multimodal extraction
- Upload utils · Uploaded file validation and normalization
Utils/upload_utils.py - Ingestion · Parses documents, images, audio, and video into usable content
Ingestion/ingestion.py
Retrieval
Embedding, indexing, search, and persistence
- Embeddings · Builds text and image embeddings for downstream search
Embeddings/embeddings.py - Indexing · Packages chunks and vectors into Qdrant records
Retrieval/indexing.py - Qdrant client · Connection and collection management for Qdrant
Retrieval/qdrant_client.py - Retriever · Queries the vector store for relevant text and visual matches
Retrieval/retrieval.py
Generation
Answer synthesis from retrieved context
Storage
Local durable Qdrant state
- rag_text · Persistent text-embedding collection
qdrant_storage/collections/rag_text - rag_image · Persistent visual-embedding collection
qdrant_storage/collections/rag_image - Qdrant store · On-disk Qdrant state including WAL, shards, and segments
qdrant_storage
Other components
- YOLOv8n · Object detection support for visual/video frame analysis
yolov8n.pt - Dependencies · Python package dependencies for UI, OCR, embeddings, and vector search
requirements.txt - README · Describes the multimodal RAG workflow and runtime setup
README.md
Connections 20
- App → Streamlit cfg: uses · UI runtime configuration
- App → Upload utils: uploads · Normalizes user-provided files
- App → Ingestion: ingests · Triggers multimodal extraction
- App → Retriever: queries · Runs question-time retrieval
- App → Generator: answers · Displays grounded responses
- Upload utils → Ingestion: prepares · Hands off validated local file paths
- Ingestion → Embeddings: extracts · Produces text and visual content for vectorization
- Ingestion → YOLOv8n: analyzes · Supports object detection on images and video frames
- Embeddings → Indexing: vectors · Provides embeddings for persistence
- Indexing → Qdrant client: writes via · Uses the Qdrant adapter for collection operations
- Qdrant client → Qdrant store: persists to · Stores and reads local Qdrant state
- Indexing → rag_text: stores · Writes text chunks and metadata
- Indexing → rag_image: stores · Writes image and visual-frame records
- Retriever → Qdrant client: reads via · Searches collections through the Qdrant adapter
- Retriever → rag_text: searches · Retrieves text-space matches
- Retriever → rag_image: searches · Retrieves image-space matches
- Retriever → Generator: supplies context · Passes ranked evidence into generation
- Generator → README: guided by · Grounded-answer behavior described in project docs
- Gen test → Generator: checks · Validates generation behavior
- Dependencies → App: supports · Declares runtime dependencies