# anjanykumarjaiswal/visiontext-llm architecture

> Architecture diagram of the GitHub repository anjanykumarjaiswal/visiontext-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anjanykumarjaiswal/visiontext-llm
- Repository: https://github.com/anjanykumarjaiswal/visiontext-llm

## Overview

##### Overall System Purpose and Flow
- That the goal is to extract text and images from PDF files, embed them into a shared vector space, perform semantic retrieval via Pinecone, and feed the results to a multimodal LLM (Google Gemini) to generate answers.
- The application is exposed as a FastAPI service (see src/main.py) that handles query requests.

##### Main Components
- API Layer: Represented by the FastAPI server (src/main.py) which exposes endpoints (e.g., /run?query=...).
- PDF Processing Component: Represented by the preprocessing module (src/utils/preprocessing.py) which extracts text and images from PDF documents using libraries like PyPDF2, Fitz, and Pillow.
- Multimodal Embedding Component: Found in src/models/embeddings.py, where the MultiModalEmbedder utilizes a Hugging Face ‘AlignModel’ to convert both texts and images into embeddings.
- Vector Database Interface: The Pinecone interaction functionalities found in src/utils/pineconedb.py – handling upserts and querying the vector store.
- Answer Generation Component: In src/models/gemini.py which holds the GenerateGeminiResponse class to process the retrieved embeddings/text/images and generate a multimodal answer using the Google Gemini API.

##### Relationships and Data Flow
- The query is processed and, if needed, triggers the PDF Processing component to extract content.
- The extracted content is passed to the Multimodal Embedding component, which generates embeddings from text and images.
- The embeddings are then upserted into or queried from Pinecone via the Database Interface.
- Finally, the retrieved relevant content is sent to the Answer Generation component, where the multimodal LLM (Google Gemini) formulates a response.
- Return the answer to the API layer and hence to the user.

##### Architectural Patterns / Design Principles
- Modular Design: The separation into distinct modules (API, preprocessing, embedding, database interaction, and answer generation) follows a clear separation of concerns.
- Pipeline Architecture: The sequential flow from document processing to embedding to retrieval, and finally, language model-based response generation.
- Use of External Services: Clearly delineate integration with external services (Hugging Face, Pinecone, Google Gemini).

##### Specify Technologies and Frameworks
- API: FastAPI powered by Uvicorn.
- Embedding: Hugging Face Transformers with the AlignModel.
- Vector Database: Pinecone.
- LLM: Google Gemini.
- PDF / Image Processing: PyPDF2, Fitz, and Pillow.
- Containerization: Docker (as hinted by Dockerfile and .dockerignore) suggests the possibility of containerized deployment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External User
    U["User Query"]:::external

    %% API Layer
    subgraph "API Layer"
        A["FastAPI Server (main.py)"]:::api
    end

    %% Data Extraction Layer
    subgraph "Data Extraction Layer"
        B["PDF Extraction (preprocessing.py)"]:::process
    end

    %% Embedding Layer
    subgraph "Embedding Layer"
        C["Multimodal Embedder (embeddings.py)"]:::process
        HF["Hugging Face AlignModel"]:::external
    end

    %% Storage/Retrieval Layer
    subgraph "Storage/Retrieval Layer"
        D["Vector Database Interface (pineconedb.py)"]:::db
        P["Pinecone Service"]:::external
    end

    %% Answer Generation Layer
    subgraph "Answer Generation Layer"
        E["Gemini Answer Generator (gemini.py)"]:::generation
        G["Google Gemini API"]:::external
    end

    %% Data Flow
    U --> A
    A --> B
    B --> C
    C --> D
    D --> E
    E --> A

    %% External Integrations
    C --"uses"--> HF
    D --"queries/upserts"--> P
    P --"responds"--> D
    E --"calls"--> G

    %% Styles for Nodes
    classDef api fill:#f8b400,stroke:#333,stroke-width:2px,color:#fff;
    classDef process fill:#4aa3df,stroke:#333,stroke-width:2px,color:#fff;
    classDef db fill:#7bdcb5,stroke:#333,stroke-width:2px,color:#fff;
    classDef generation fill:#ee4b2b,stroke:#333,stroke-width:2px,color:#fff;
    classDef external fill:#cfcfcf,stroke:#333,stroke-width:2px,color:#000;

    %% Click Events for Component Mapping
    click A "https://github.com/anjanykumarjaiswal/visiontext-llm/blob/main/src/main.py"
    click B "https://github.com/anjanykumarjaiswal/visiontext-llm/blob/main/src/utils/preprocessing.py"
    click C "https://github.com/anjanykumarjaiswal/visiontext-llm/blob/main/src/models/embeddings.py"
    click D "https://github.com/anjanykumarjaiswal/visiontext-llm/blob/main/src/utils/pineconedb.py"
    click E "https://github.com/anjanykumarjaiswal/visiontext-llm/blob/main/src/models/gemini.py"
```
