# topping1/bm25-pdf-search architecture

> Architecture diagram of the GitHub repository topping1/bm25-pdf-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/topping1/bm25-pdf-search
- Repository: https://github.com/topping1/bm25-pdf-search

## Overview

##### Input Layer
- PDF Files Storage
- JSON Files Storage (preprocessed PDF content)
- Search Query Interface (PyQt5 GUI)

##### Processing Layer
- PDF Text Extraction Module (PyMuPDF4LLM)
- Text Chunking System
- BM25 Search Engine (BM25s)

##### Layer
- Text Display Component
- PDF Viewer Component
- Navigation Controls

##### Component Relationships
a. PDF Files → Text Extraction → JSON Storage
b. User Query → BM25 Search Engine → JSON Files
c. Search Results → Display Components

##### Specific Implementation Details
- Two main processes:
1. Preprocessing Pipeline (chunk-pdf-pages.py):
PDF Files → PyMuPDF4LLM → Page Chunking → JSON Storage
2. Search Pipeline (BM25-PDF-Search.py):
Query → BM25 → Results → Dual Display (Text + PDF)

##### Specific elements
- PDF File Repository
- JSON Cache Storage
- PyQt5 GUI Components
- BM25 Search Engine
- Text Extraction Module
- Synchronized Display System
- User Input Handler
- Navigation Controller

##### Technical Notes
- That it's a file-system based application
- That JSON files mirror PDF files (1:1 relationship)
- Synchronization between text and PDF views
- Search keyword highlighting system

- The simplicity of the architecture (no client-server, no database)
- The two-phase nature of the system (preprocessing and search)
- The dual-view presentation system
- The file-based storage approach

- Data flow directions
- File storage types
- Processing components
- User interface elements
- External dependencies

This will provide a clear understanding of how the system processes PDFs, stores the extracted text, performs searches, and presents results to users.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Define styles
    classDef storage fill:#b3d1ff,stroke:#333,stroke-width:2px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:2px
    classDef ui fill:#FFB366,stroke:#333,stroke-width:2px
    classDef external fill:#E0E0E0,stroke:#333,stroke-width:2px

    %% Input/Storage Layer
    subgraph InputLayer["Input & Storage Layer"]
        PDFFiles["PDF Files Repository"]:::storage
        JSONCache["JSON Cache Storage"]:::storage
    end

    %% Processing Layer
    subgraph ProcessingLayer["Processing Layer"]
        direction TB
        PreprocessingPipeline["Preprocessing Pipeline"]:::processing
        SearchPipeline["Search Pipeline"]:::processing
        TextExtraction["Text Extraction Module"]:::processing
        ChunkingSystem["Text Chunking System"]:::processing
        BM25Engine["BM25 Search Engine"]:::processing
    end

    %% Display Layer
    subgraph DisplayLayer["Display Layer"]
        direction TB
        GUI["PyQt5 GUI"]:::ui
        TextDisplay["Text Display Component"]:::ui
        PDFViewer["PDF Viewer Component"]:::ui
        NavControls["Navigation Controls"]:::ui
    end

    %% External Dependencies
    PyMuPDF["PyMuPDF4LLM"]:::external

    %% Relationships
    PDFFiles --> PreprocessingPipeline
    PreprocessingPipeline --> TextExtraction
    TextExtraction --> ChunkingSystem
    ChunkingSystem --> JSONCache
    PyMuPDF --> TextExtraction

    GUI --> SearchPipeline
    SearchPipeline --> BM25Engine
    BM25Engine --> JSONCache
    BM25Engine --> TextDisplay
    BM25Engine --> PDFViewer
    NavControls --> TextDisplay
    NavControls --> PDFViewer

    %% Click events for component mapping
    click PreprocessingPipeline "https://github.com/Topping1/BM25-PDF-Search/blob/main/chunk-pdf-pages.py"
    click SearchPipeline "https://github.com/Topping1/BM25-PDF-Search/blob/main/BM25-PDF-Search.py"

    %% Legend
    subgraph Legend
        StorageL["Storage Component"]:::storage
        ProcessingL["Processing Component"]:::processing
        UIL["UI Component"]:::ui
        ExternalL["External Dependency"]:::external
    end
```
