# sanjith-3/multimodal-rag-gemini-fastapi architecture

> Architecture diagram of the GitHub repository sanjith-3/multimodal-rag-gemini-fastapi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanjith-3/multimodal-rag-gemini-fastapi
- Repository: https://github.com/sanjith-3/multimodal-rag-gemini-fastapi

## Overview

##### Main Components
a. User Interface (Frontend)
- Streamlit Chat App (API/Streamlit_Chat_app.py)
b. Backend Service (API Layer)
- FastAPI endpoints (API/rag_chat.py or API/rag_api.py)
- Upload endpoint (/upload-doc)
- Chat endpoint (/chat)
- Evaluation endpoint (/evaluate)
c. Document Processing Pipeline
- UnstructuredPDFLoader (text + image extraction)
- TextProcessor (chunking, summarization)
- ImageProcessor (image description)
d. Retrieval & Storage
- ChromaDB vector store (API/chromadb_storage/chroma.sqlite3)
- ChromaMultiModalRetriever
e. RAG Orchestrator
- MultiModalRAG.query() (assembles prompt, retrieves context, calls Gemini)
f. External Services
- Gemini Multimodal API (via `google-generativeai`)
- Poppler & Tesseract (local PDF/OCR)
g. Evaluation Module
- BLEU/ROUGE (sentence-transformers)
- Image similarity (Gemini vision)
- Response relevancy (API/evaluation.py)
h. Supporting Artifacts
- Upload storage (API/uploads)
- Extracted images (API/figures)
- Notebooks (development, demos)
- Test_Cases (validation scenarios)

##### Relationships & Data Flows
- User → Streamlit UI: uploads PDF, enters query
- Streamlit → FastAPI: HTTP POST to /upload-doc and /chat
- FastAPI → Document Pipeline: load & process PDFs → images & text
- Processed data → ChromaDB: embeddings stored
- FastAPI /chat → Retriever: fetch top‐k text/image embeddings
- Retriever → RAG Orchestrator: build multimodal prompt
- RAG Orchestrator → Gemini API: send mixed text+image query
- Gemini → RAG Orchestrator → FastAPI → Streamlit: return answer + sources
- FastAPI /evaluate → Evaluation Module: compute metrics → return scores

##### Architectural Patterns
- Separation of Concerns:
- UI layer (Streamlit)
- API layer (FastAPI)
- Processing layer (PDF extraction, embedding)
- Storage layer (ChromaDB)
- External integration (Gemini, OCR tools)
- Retrieval-Augmented Generation (RAG):
- Retrieval sub‐pipeline
- Generation sub‐pipeline
- Modularity for Evaluation:
- Plug-in metrics module

##### Technologies & Frameworks to Annotate
- FastAPI (web framework)
- Streamlit (UI)
- ChromaDB (vector DB)
- google-generativeai (Gemini)
- UnstructuredPDFLoader, pdf2image, Tesseract (PDF/OCR)
- sentence-transformers (embeddings)
- poppler-utils (PDF support)

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Frontend Layer"
        UI["Streamlit Chat App"]:::frontend
    end

    subgraph "API Layer (FastAPI)" 
        upload["POST /upload-doc"]:::api
        chat["POST /chat"]:::api
        evalEP["POST /evaluate"]:::api
        ragapi["rag_api.py"]:::api
        ragchat["rag_chat.py"]:::api
    end

    subgraph "Document Processing Pipeline"
        pdfloader["UnstructuredPDFLoader & OCR"]:::processing
        textproc["TextProcessor"]:::processing
        imgproc["ImageProcessor"]:::processing
    end

    subgraph "Storage Layer"
        chromadir["ChromaDB Directory"]:::storage
        chromadb["chroma.sqlite3"]:::storage
        uploads["Uploads Directory"]:::storage
        figures["Figures Directory"]:::storage
    end

    subgraph "RAG Orchestrator"
        orchestrator["MultiModalRAG.query()"]:::backend
    end

    subgraph "External Services"
        gemini["Gemini Multimodal API"]:::external
        ocrtools["Poppler & Tesseract"]:::external
    end

    subgraph "Evaluation Module"
        evaluation["Evaluation Module"]:::evaluation
    end

    subgraph "Supporting Artifacts"
        demo1["Final_Multimodal_RAG.ipynb"]:::docs
        demo2["demo_rag.ipynb"]:::docs
        tc1["High/text.txt"]:::docs
        tc2["Neutral/text.txt"]:::docs
        tc3["Low/text.txt"]:::docs
    end

    UI -->|"POST /upload-doc"| upload
    UI -->|"POST /chat"| chat

    upload --> pdfloader
    pdfloader --> textproc
    pdfloader --> imgproc
    textproc --> chromadb
    imgproc --> chromadb

    chat --> orchestrator
    orchestrator --> chromadb
    orchestrator --> gemini
    gemini --> orchestrator
    orchestrator --> chat

    evalEP -.->|"compute metrics"| evaluation
    evaluation --> evalEP

    classDef frontend fill:#ADD8E6,stroke:#000
    classDef api fill:#90EE90,stroke:#000
    classDef processing fill:#FDEBD0,stroke:#000
    classDef storage fill:#FFB6C1,stroke:#000
    classDef backend fill:#98FB98,stroke:#000
    classDef external fill:#FFA500,stroke:#000
    classDef evaluation fill:#DA70D6,stroke:#000
    classDef docs fill:#D3D3D3,stroke:#000

    click UI "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/API/Streamlit_Chat_app.py"
    click ragapi "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/API/rag_api.py"
    click ragchat "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/API/rag_chat.py"
    click chromadir "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/tree/main/API/chromadb_storage"
    click chromadb "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/API/chromadb_storage/chroma.sqlite3"
    click uploads "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/tree/main/API/uploads"
    click figures "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/tree/main/API/figures"
    click evaluation "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/API/evaluation.py"
    click demo1 "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/Notebooks/Final_Multimodal_RAG.ipynb"
    click demo2 "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/Notebooks/demo_rag.ipynb"
    click tc1 "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/Test Cases/High/text.txt"
    click tc2 "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/Test Cases/Neutral/text.txt"
    click tc3 "https://github.com/sanjith-3/multimodal-rag-gemini-fastapi/blob/main/Test Cases/Low/text.txt"
```
