How dubeamit/multimodal-rag works
This is a multimodal RAG application for uploading PDFs and videos, asking grounded questions across a session’s files, and navigating answers back to cited pages or timestamps. The main workflow runs from the browser chat through FastAPI ingestion or querying, then through modality-specific extraction and session-scoped vector retrieval to a local or Gemini model. The interface persists sessions locally and routes citations to its PDF viewer or video player. Some frontend upload and citation-rendering details are only partially sampled; those links follow the README and visible component interfaces.
Main entry points: page.tsx Session and media pane · main.py FastAPI endpoints · vector_store.py Chunk indexing and RAG · ChatInterface.tsx Chat and citations
Components 11
Browser experience
- Session and media pane
frontend/src/app/page.tsx - Chat and citations
frontend/src/components/ChatInterface.tsx - PDF viewer
frontend/src/components/PdfViewer.tsx
API orchestration
- FastAPI endpoints
backend/main.py
Document ingestion
- PDF and video processing
backend/multimodal_processor.py - Audio transcription runner
backend/speech2text/transcribe.sh
Retrieval and generation
- Chunk indexing and RAG
backend/vector_store.py - Session vector collection
backend/vector_store.py
Other components
- User
- Local OpenAI-compatible model
- Gemini model
Connections 14
- User → Session and media pane: uses
- Session and media pane → Chat and citations: renders
- Chat and citations → FastAPI endpoints: uploads or queries
- FastAPI endpoints → PDF and video processing: processes files
- FastAPI endpoints → Chunk indexing and RAG: indexes chunks
- PDF and video processing → Audio transcription runner: runs transcription
- FastAPI endpoints → Chunk indexing and RAG: queries RAG
- Chunk indexing and RAG → Session vector collection: embeds, stores, retrieves
- Chunk indexing and RAG → Local OpenAI-compatible model: requests answer
- Chunk indexing and RAG → Gemini model: requests answer
- FastAPI endpoints → Chat and citations: returns answer and citations
- Chat and citations → Session and media pane: dispatches citation
- Session and media pane → PDF viewer: sets cited page
- Session and media pane → Session and media pane: seeks video timestamp