How akshitkhare07/multimodal_rag works
What this project is (type + purpose)
This is a single-process Python Streamlit application implementing multimodal RAG:
• Inputs: documents (PDF/TXT/CSV/DOCX/XLSX/JSON) and videos (MP4/MOV/…).
• Processing: extract text (and for video: audio → transcription via Whisper).
• Indexing: chunk text + create embeddings; store in FAISS vector index.
• Retrieval + generation: semantic search over FAISS; pass retrieved context to Groq LLM for answers.
• Persistence: “session management” to save/load vector stores (likely local disk).
So architecturally it’s a monolithic interactive app with clear internal modules and a couple of external dependencies (Groq, Whisper/FFmpeg).
What the file tree tells you to model (modules → components)
From /src:
• app.py — presentation + orchestration (Streamlit UI, user flows, session state).
• data_loader.py — document ingestion (parsing/extraction per file type).
• video_loader.py — video ingestion (audio extraction + Whisper transcription).
• ChunkAndEmbed.py — text chunking + embedding pipeline (core indexing transform).
• vectorStore.py — FAISS index lifecycle (create/update/query/save/load).
• search.py — RAG query path (retrieval + Groq prompt/LLM call + formatting/sources).
Other notable runtime/config:
• .streamlit/config.toml — Streamlit config (UI/server behavior).
• .streamlit/secrets.toml (described in README) — GROQ_API_KEY.
• data/ — sample inputs (test PDF, test MP4).
• DEPLOYMENT_NOTES.md, STREAMLIT_CLOUD_SETUP.md — likely indicate deployment model is Streamlit Cloud (still single service).
Ingestion Layer
• Document Loader (data_loader.py)
• Video Loader (video_loader.py)
Indexing Pipeline
• Chunk & Embed (ChunkAndEmbed.py)
Vector Store Service
• FAISS Store Manager (vectorStore.py)
RAG Query Service
• Retrieval + Groq LLM (search.py)
External connectors
• Groq LLM API
• Whisper transcription engine (and FFmpeg if applicable)
• Filesystem persistence
app.py calls video_loader.py
3. video_loader.py extracts audio (optionally via FFmpeg) → runs Whisper → transcript (+ timestamps if available)
Transcript goes to ChunkAndEmbed.py → embeddings
5. vectorStore.py upserts into FAISS with metadata source_type=video, possibly time ranges
Sequence 4 (optional but important): Save/Load session
• app.py → vectorStore.py.save() writes FAISS index + metadata to disk
• app.py → vectorStore.py.load() restores index into session
Main components to include (and how to label them)
Use these component blocks (separate as much as possible):
Ingestion
• Document Parser (src/data_loader.py)
• Video Processor (src/video_loader.py)
• Audio extraction
• Whisper transcription
Indexing
• Chunker (src/ChunkAndEmbed.py)
• If it’s a hosted embedding API, show it as external; if local, show it as internal library.
Storage / Retrieval
• FAISS Vector Index (src/vectorStore.py)
• Local Persistence
• Saved FAISS files + metadata (session save/load)
Generation
• RAG Orchestrator (src/search.py)
• Groq LLM API (external)
• Upload → app.py: “file bytes + filename + user action”
• app.py → data_loader.py: “document file → extracted text + metadata”
• app.py → video_loader.py: “video file → transcript (+ timestamps)”
• Loader → ChunkAndEmbed.py: “raw text”
• ChunkAndEmbed.py → vectorStore.py: “(chunk_id, embedding[], chunk_text, metadata)”
• search.py → vectorStore.py: “query embedding → topK chunks”
• search.py → Groq: “prompt(context + question)”
• Groq → search.py: “answer text”
• vectorStore.py ↔ Disk: “save/load index + metadata”
• app.py ↔ search.py: “question + filters → answer + sources”
Architectural patterns worth calling out
• Pipeline pattern for ingestion/indexing: Load → Normalize text → Chunk → Embed → Upsert.
• RAG pattern for query: Embed(query) → Retrieve → Compose prompt → Generate.
• Session-scoped state: vector store is “temporary during the session” but can be persisted via save/load.
What “very detailed” means for this repo (don’t overthink—split components)
To align with “separating into as many components as possible,” break out these subcomponents even if they live in one file:
• Document loaders by format (PDF vs DOCX vs CSV)
• Video: audio extraction vs transcription
• Chunking vs embedding
• VectorStore: index build/upsert vs query vs persistence
• Search: retrieval vs prompt construction vs LLM call vs citation formatting
That level of detail will produce the most accurate architecture representation for a principal-engineer audience and makes bottlenecks and boundaries immediately visible.