How utkarshk0509/multimodal-rag works
This is a Streamlit multimodal document RAG application. Its primary workflow accepts a PDF, extracts native text or uses Gemini vision for scanned pages, chunks and indexes content in Chroma, then accepts text or voice questions and returns Gemini answers grounded in retrieved passages. Separate scripts support satellite-document parsing, chart description, index initialization, and command-line retrieval. The README provides no additional capability details; relationships shown come from the supplied source excerpts.
Main entry points: app.py Streamlit app · generate_rag_response.py CLI RAG response · vector_store_init.py Index initializer · vision_extractor.py Chart description
Components 18
Document Ingestion
- Streamlit app
app.py - PDF extraction
app.py - Vision OCR
app.py - Text chunking
app.py
Index and Retrieval
- Chroma store
- Index initializer
vector_store/vector_store_init.py - Filtered retrieval
retrieval/query_rag.py
Question Answering
- Voice and text input
app.py - Context retrieval
app.py - RAG prompt
app.py - Answer generation
app.py - CLI RAG response
retrieval/generate_rag_response.py
Supporting Pipelines
- Satellite PDF parser
data_ingestion/advanced_parser.py - PDF preview utility
data_ingestion/pdf_parser.py - Chart description
data_ingestion/vision_extractor.py
Other components
- Document user
- Google Gemini
- Microphone transcription
Connections 21
- Document user → Streamlit app: uploads PDF
app.py - Streamlit app → PDF extraction: processes PDF
app.py - PDF extraction → Vision OCR: OCR fallback
app.py - Vision OCR → Google Gemini: generates OCR
app.py - Streamlit app → Text chunking: chunks pages
app.py - Text chunking → Google Gemini: embeds documents
app.py - Text chunking → Chroma store: persists chunks
app.py - Document user → Voice and text input: asks question
app.py - Voice and text input → Microphone transcription: transcribes audio
app.py - Voice and text input → Context retrieval: submits query
app.py - Context retrieval → Chroma store: retrieves context
app.py - Context retrieval → RAG prompt: formats context
app.py - RAG prompt → Answer generation: supplies prompt
app.py - Answer generation → Google Gemini: generates answer
app.py - Answer generation → Document user: displays answer
app.py - Satellite PDF parser → Chart description: writes images
data_ingestion/advanced_parser.py - Chart description → Index initializer: writes descriptions
data_ingestion/vision_extractor.py - Index initializer → Chroma store: indexes documents
vector_store/vector_store_init.py - Filtered retrieval → Chroma store: retrieves profiles
retrieval/query_rag.py - CLI RAG response → Chroma store: retrieves context
retrieval/generate_rag_response.py - CLI RAG response → Google Gemini: generates answer
retrieval/generate_rag_response.py