# saumitras/colpali-milvus-multimodal-rag architecture

> Architecture diagram of the GitHub repository saumitras/colpali-milvus-multimodal-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saumitras/colpali-milvus-multimodal-rag
- Repository: https://github.com/saumitras/colpali-milvus-multimodal-rag

## Overview

##### Main Components
- Frontend / API Service:
- app.py – likely the main entry point that manages HTTP requests, user interactions (e.g., PDF uploads), and routes queries.
- PDF Processing:
- pdf_manager.py – handles PDF ingestion and conversion into an image representation.
- Embedding Extraction:
- colpali_manager.py – manages interfacing with the Colpali model for generating page embeddings.
- Vector Database Indexing:
- milvus_manager.py – responsible for indexing the embeddings into Milvus for retrieval.
- Query Handling / RAG Logic:
- rag.py – contains logic for coordinating the retrieval process; it uses the indexed embeddings to facilitate answer generation.
- Middleware / Utility Functions:
- middleware.py and utils.py – implement cross-cutting concerns, such as request handling, error management or common utilities.
- External Dependencies and Configuration:
- requirements.txt and packages.txt – list the libraries and packages used to run the project.

##### Relationships and Data Flow
- The PDF is processed by pdf_manager.py, which treats the document as an image.
- The processed data is sent to colpali_manager.py to compute embeddings.
- These embeddings are then forwarded to milvus_manager.py, where they are indexed and stored in Milvus.
- When a search or Q&A query is issued, the system uses rag.py to retrieve relevant embeddings from Milvus.
- The retrieved results are then passed to the visual LLM (represented externally, e.g., gemini/gpt-4o) to generate the final answer.
- Middleware and utils support and streamline these interactions—ensuring a smooth data flow and request lifecycle.

##### Architectural Patterns and Design Principles
- Clear Separation of Concerns: Each module is designed for a distinct responsibility (PDF processing, embedding extraction, indexing, query handling), which helps maintain modularity and scalability.
- Microservices/Modular Architecture: Although all modules are part of a single codebase, the clear separation suggests you could evolve or scale individual components independently.
- Integration of External Services: Highlight the connection with external systems (Milvus for indexing, Colpali for embeddings, visual LLM for query answering).

- Connection and two-way interaction between rag.py and milvus_manager.py (indicating query retrieval).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Entities
    User["User Interface"]:::external

    %% Core Modules
    app["API Service"]:::api
    pdf["PDF Manager"]:::processing
    colp["Colpali Embedding Engine"]:::processing
    milvus["Vector Database Index"]:::indexing
    rag["RAG Query Processor"]:::query
    llm["Visual LLM"]:::external

    %% Supporting Components
    middleware["Middleware"]:::support
    utils["Utils"]:::support
    req["Requirements"]:::config
    pkg["Packages"]:::config

    %% Data Flow - Upload and Processing Pipeline
    User -->|"uploads PDF"| app
    app -->|"process PDF"| pdf
    pdf -->|"convert to embeddings"| colp
    colp -->|"store embeddings"| milvus

    %% Data Flow - Query Pipeline
    app -->|"receives query"| rag
    rag -- "retrieve" --- milvus
    milvus -- "return results" --- rag
    rag -->|"forward query"| llm

    %% Supporting Flow
    middleware --- app
    utils --- app
    req --- app
    pkg --- app

    %% Click Events
    click app "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/app.py"
    click pdf "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/pdf_manager.py"
    click colp "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/colpali_manager.py"
    click milvus "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/milvus_manager.py"
    click rag "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/rag.py"
    click middleware "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/middleware.py"
    click utils "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/utils.py"
    click req "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/requirements.txt"
    click pkg "https://github.com/saumitras/colpali-milvus-multimodal-rag/blob/master/packages.txt"

    %% Styles
    classDef api fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef processing fill:#A9DFBF,stroke:#27AE60,stroke-width:2px;
    classDef indexing fill:#F7DC6F,stroke:#F1C40F,stroke-width:2px;
    classDef query fill:#D7BDE2,stroke:#8E44AD,stroke-width:2px;
    classDef external fill:#FADBD8,stroke:#E74C3C,stroke-width:2px;
    classDef support fill:#D5DBDB,stroke:#566573,stroke-width:2px;
    classDef config fill:#BB8FCE,stroke:#6C3483,stroke-width:2px;
```
