How sanjith-3/multimodal-rag-gemini-fastapi works
Main Components
a. User Interface (Frontend)
• Streamlit Chat App (API/Streamlit_Chat_app.py)
b. Backend Service (API Layer)
• FastAPI endpoints (API/rag_chat.py or API/rag_api.py)
• Upload endpoint (/upload-doc)
• Chat endpoint (/chat)
• Evaluation endpoint (/evaluate)
c. Document Processing Pipeline
• UnstructuredPDFLoader (text + image extraction)
• TextProcessor (chunking, summarization)
• ImageProcessor (image description)
d. Retrieval & Storage
• ChromaDB vector store (API/chromadb_storage/chroma.sqlite3)
• ChromaMultiModalRetriever
e. RAG Orchestrator
• MultiModalRAG.query() (assembles prompt, retrieves context, calls Gemini)
f. External Services
• Gemini Multimodal API (via google-generativeai)
• Poppler & Tesseract (local PDF/OCR)
g. Evaluation Module
• BLEU/ROUGE (sentence-transformers)
• Image similarity (Gemini vision)
• Response relevancy (API/evaluation.py)
h. Supporting Artifacts
• Upload storage (API/uploads)
• Extracted images (API/figures)
• Notebooks (development, demos)
• Test_Cases (validation scenarios)
Relationships & Data Flows
• User → Streamlit UI: uploads PDF, enters query
• Streamlit → FastAPI: HTTP POST to /upload-doc and /chat
• FastAPI → Document Pipeline: load & process PDFs → images & text
• Processed data → ChromaDB: embeddings stored
• FastAPI /chat → Retriever: fetch top‐k text/image embeddings
• Retriever → RAG Orchestrator: build multimodal prompt
• RAG Orchestrator → Gemini API: send mixed text+image query
• Gemini → RAG Orchestrator → FastAPI → Streamlit: return answer + sources
• FastAPI /evaluate → Evaluation Module: compute metrics → return scores
Architectural Patterns
• Separation of Concerns:
• UI layer (Streamlit)
• API layer (FastAPI)
• Processing layer (PDF extraction, embedding)
• Storage layer (ChromaDB)
• External integration (Gemini, OCR tools)
• Retrieval-Augmented Generation (RAG):
• Retrieval sub‐pipeline
• Generation sub‐pipeline
• Modularity for Evaluation:
• Plug-in metrics module
Technologies & Frameworks to Annotate
• FastAPI (web framework)
• Streamlit (UI)
• ChromaDB (vector DB)
• google-generativeai (Gemini)
• UnstructuredPDFLoader, pdf2image, Tesseract (PDF/OCR)
• sentence-transformers (embeddings)
• poppler-utils (PDF support)