How aranya-chatterjee/multimodal_rag works
1. Overall shape
- This is a single-process Streamlit application that acts as a multimodal RAG front end.
- The main flow is: upload file → extract text/transcript → chunk and embed → store in FAISS → search/retrieve → send context to Groq LLM → show answer and sources.
2. UI / orchestration layer
- src/app.py is the top-level application entry point and likely owns the Streamlit sidebar/main-page interaction.
- It orchestrates file upload, source type selection, session actions like saving/loading, and user queries.
- The UI is the only user-facing boundary; all processing happens in the supporting modules.
3. Document ingestion subsystem
- src/data_loader.py handles non-video inputs.
- Per the README, it supports PDF, TXT, CSV, DOCX, XLSX, and JSON, so this module is the document normalization boundary.
- Its output is presumably plain text or document chunks suitable for downstream embedding.
4. Video ingestion subsystem
- src/video_loader.py handles MP4/MOV/AVI/MKV/WMV inputs.
- The README says it performs audio transcription, so the video path is effectively: video file → extracted audio/transcript → text for RAG.
- This is the main multimodal boundary in the repo: video content is converted into searchable text before entering the same retrieval pipeline as documents.
5. Chunking and embedding
- src/ChunkAndEmbed.py is the text preparation layer.
- It likely splits loaded text/transcripts into chunks and generates embeddings.
- This module is the bridge between raw content and retrieval-ready vector representations, so it is central to both document and video ingestion.
6. Vector storage
- src/vectorStore.py manages the FAISS-based vector store mentioned in the README.
- This is the persistence/search index layer for embeddings and associated metadata.
- The repository treats the vector store as session-scoped state with optional save/load behavior, rather than a long-lived external database.
7. Retrieval and answer generation
- src/search.py performs semantic search over the vector store and connects retrieval results to the Groq LLM.
- This module is the RAG “reasoning” boundary: it takes a user question, retrieves relevant chunks, and produces an answer grounded in the retrieved context.
- It also appears to surface source references, which means retrieved chunks are preserved with enough metadata to cite them back in the UI.
8. Data flow for document questions
- For documents, the path is: upload in Streamlit → data_loader.py parses content → ChunkAndEmbed.py creates chunks and embeddings → vectorStore.py stores them → search.py retrieves relevant chunks → Groq generates the response.
- This is a standard text-only RAG flow.
9. Data flow for video questions
- For videos, the path is: upload in Streamlit → video_loader.py transcribes audio → the transcript goes through the same chunk/embed/store/search pipeline.
- The important architectural point is that video is not searched directly; it is normalized into text first, then merged into the same retrieval index as documents.
10. State and lifecycle
- The app appears to keep the vector store in memory during the Streamlit session and optionally persist it to disk via “Save Session.”
- That makes the system lightweight and self-contained, but it also means retrieval state is application-managed rather than backed by a database service.
11. External dependencies that matter architecturally
- Groq API is the external LLM runtime used for generation.
- Whisper is used for video transcription, which is why video ingestion is slower and more compute-heavy than document ingestion.
- FAISS is the local vector search engine, so semantic retrieval is done in-process rather than via a hosted search service.
12. Configuration and deployment boundaries
- .streamlit/config.toml and .streamlit/secrets.toml are important operational files.
- The app depends on a secret GROQ_API_KEY, so the LLM integration is environment-configured rather than hardcoded.
- The presence of DEPLOYMENT_NOTES.md and STREAMLIT_CLOUD_SETUP.md suggests the app is intended to run locally or on Streamlit Cloud with minimal infrastructure.
13. Repository structure summary
- src/app.py = orchestration/UI.
- src/data_loader.py + src/video_loader.py = ingestion and normalization.
- src/ChunkAndEmbed.py = chunking/embedding transformation.
- src/vectorStore.py = vector index management.
- src/search.py = retrieval + LLM answer synthesis.
- data/ contains sample inputs for testing the pipeline end to end.
14. How to diagram it
- Draw a single Streamlit UI at the top, with two ingestion branches: documents and videos.
- Merge both branches into a shared chunk/embed stage, then into a FAISS vector store.
- From the query side, show user questions flowing into search/retrieval, then into Groq for response generation, and finally back to the UI with answers plus source references.
Main entry points: app.py Streamlit app · search.py RAG answer · vectorStore.py FAISS store · video_loader.py Video loader
Components 16
UI & Orchestration
Streamlit entry point and session flow
- Streamlit app · Top-level UI and orchestration for uploads, queries, and session actions
src/app.py - Controls · Source selection, upload, and session management interactions
src/app.py - Answer view · Displays responses, sources, and current session state
src/app.py - User · Uploads content and asks questions through the Streamlit interface
Ingestion
Document and video normalization into text
- Doc loader · Normalizes PDF, TXT, CSV, DOCX, XLSX, and JSON inputs into text
src/data_loader.py - Video loader · Extracts audio and transcribes video files into text
src/video_loader.py
Retrieval & Generation
Chunking, vector search, and answer synthesis
- Chunker · Splits normalized text into retrieval-sized chunks
src/ChunkAndEmbed.py - Embedder · Creates vector representations for chunks and transcripts
src/ChunkAndEmbed.py - FAISS store · In-process vector store for saving and searching embedded chunks
src/vectorStore.py - Store ops · Session save/load and index lifecycle management
src/vectorStore.py - Search · Semantic retrieval over the vector store for user questions
src/search.py - RAG answer · Combines retrieved context with the question to generate a grounded answer
src/search.py - Sources · Carries chunk metadata used to show references back in the UI
src/search.py
External Services
Third-party runtime dependencies
- Groq API · External model provider used for response generation
- Whisper · External transcription runtime used for video audio extraction
- FAISS · Local nearest-neighbor search engine backing semantic retrieval
Connections 19
- User → Streamlit app: uploads / asks · User interacts with the Streamlit app
- Streamlit app → Controls: routes actions · App coordinates upload, source selection, and session controls
- Streamlit app → Doc loader: document input · Document files are normalized into text
- Streamlit app → Video loader: video input · Video files are sent for transcription
- Video loader → Whisper: transcribes · Audio is converted to transcript text
- Doc loader → Chunker: raw text · Parsed documents feed the chunking stage
- Video loader → Chunker: transcript text · Transcripts join the same chunking pipeline
- Chunker → Embedder: chunks · Text chunks are embedded for retrieval
- Embedder → FAISS store: vectors · Embeddings and metadata are stored for later search
- FAISS store → FAISS: index backend · FAISS provides local vector similarity search
- Streamlit app → Store ops: save / load · Session state can be persisted and restored
- Store ops → FAISS store: manages · Persistence layer owns the vector index lifecycle
- User → Search: questions · User queries enter retrieval through the app
- Search → FAISS store: retrieve · Search pulls relevant chunks from the vector index
- Search → RAG answer: context · Retrieved passages are passed into answer synthesis
- RAG answer → Groq API: generates · Groq produces the final grounded response
- RAG answer → Sources: attaches citations · Source metadata is preserved for display
- RAG answer → Answer view: renders · Answer and sources are shown in the UI
- Answer view → User: response · Final output returns to the user