# gaurav4050/pdfreader2 architecture

> Architecture diagram of the GitHub repository gaurav4050/pdfreader2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gaurav4050/pdfreader2
- Repository: https://github.com/gaurav4050/pdfreader2

## Overview

##### Project type and purpose
- This is a Python-based backend service (FastAPI) for document ingestion, embedding, vector storage, semantic search, reranking, and summarization using an external LLM (Gemini).
- It provides APIs (via uvicorn main:app) to load documents, generate embeddings, query a vector store, rerank hits, and produce summaries.

##### File structure
- Top level:
- main.py – FastAPI entry point
- embedding/ – code to generate embeddings for text chunks
- ingest/ – ingestion scripts (e.g., ingest from Google Drive)
- vectorstore/ – Milvus client wrapper
- utils/ – helper modules (query embedding, grouping, hashing, PDF parsing, reranking, summarization)
- ground_truth.json – evaluation/reference data
- No frontend; single backend service.
- Configuration handled in code; external dependencies: Milvus, LLM API.

##### Insights from README
- Minimal: runs with “uvicorn main:app --reload”.
- Implies FastAPI with endpoints defined in main.py.

##### Detail
- Every util module as a subcomponent if desired (e.g., PDF parsing, hashing, grouping).
- Ground_truth.json only if illustrating evaluation flows.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    Files["PDF/Text Files"]:::data
    DriveIngest["Drive Folder Ingest"]:::internal
    GroundTruth["Ground Truth JSON"]:::data

    %% Data Ingestion Layer
    subgraph "Data Ingestion Layer"
        PDFUtils["PDF/Text Extraction Utility"]:::internal
        Group["Document Chunk Grouping"]:::internal
    end

    %% Embedding Layer
    subgraph "Embedding Layer"
        EmbService["Embedding Generation Service"]:::internal
    end

    %% Vector Store
    subgraph "Vector Store"
        Milvus["Milvus DB"]:::db
    end

    %% API Layer
    subgraph "API Layer"
        API["FastAPI Server"]:::internal
    end

    %% Business Logic Layer
    subgraph "Business Logic Layer"
        SearchChunks["Chunk Search Helper"]:::internal
        Hash["Hashing Utilities"]:::internal
        Rerank["Reranking (Gemini LLM)"]:::externalApi
        Summarize["Summarization (Gemini LLM)"]:::externalApi
        SummarizeKey["Keyword Summarization (Gemini LLM)"]:::externalApi
    end

    %% Client
    Client["Client"]:::data

    %% Ingestion Flow
    Files --> DriveIngest
    DriveIngest --> PDFUtils
    PDFUtils --> Group
    Group --> EmbService
    EmbService --> Milvus

    %% Query Flow
    Client --> API
    API --> QueryEmb["Query Embedding Utility"]:::internal
    QueryEmb --> Milvus
    Milvus --> SearchChunks
    SearchChunks --> Rerank
    Rerank --> Summarize
    SearchChunks --> SummarizeKey
    Summarize --> API
    SummarizeKey --> API

    %% Optional Evaluation
    GroundTruth -.-> API

    %% Click Events
    click DriveIngest "https://github.com/gaurav4050/pdfreader2/blob/master/ingest/drive_folder_ingest.py"
    click PDFUtils "https://github.com/gaurav4050/pdfreader2/blob/master/utils/pdf_utils.py"
    click Group "https://github.com/gaurav4050/pdfreader2/blob/master/utils/group_by_file_id.py"
    click EmbService "https://github.com/gaurav4050/pdfreader2/blob/master/embedding/generator.py"
    click QueryEmb "https://github.com/gaurav4050/pdfreader2/blob/master/utils/get_query_embedding.py"
    click Milvus "https://github.com/gaurav4050/pdfreader2/blob/master/vectorstore/milvus_client.py"
    click API "https://github.com/gaurav4050/pdfreader2/blob/master/main.py"
    click Rerank "https://github.com/gaurav4050/pdfreader2/blob/master/utils/rerank_results_with_gemini.py"
    click Summarize "https://github.com/gaurav4050/pdfreader2/blob/master/utils/summarize_results_with_gemini.py"
    click SummarizeKey "https://github.com/gaurav4050/pdfreader2/blob/master/utils/summarize_keyword_results_with_gemini.py"
    click SearchChunks "https://github.com/gaurav4050/pdfreader2/blob/master/utils/serach_chunks.py"
    click Hash "https://github.com/gaurav4050/pdfreader2/blob/master/utils/hash_utils.py"
    click GroundTruth "https://github.com/gaurav4050/pdfreader2/blob/master/ground_truth.json"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000;
    classDef externalApi fill:#FFA500,stroke:#000;
    classDef data fill:#90EE90,stroke:#000,stroke-dasharray: 5 5;
    classDef db fill:#FFD700,stroke:#000;
```
