How topping1/bm25-pdf-search works
Input Layer
• PDF Files Storage
• JSON Files Storage (preprocessed PDF content)
• Search Query Interface (PyQt5 GUI)
Processing Layer
• PDF Text Extraction Module (PyMuPDF4LLM)
• Text Chunking System
• BM25 Search Engine (BM25s)
Layer
• Text Display Component
• PDF Viewer Component
• Navigation Controls
Component Relationships
a. PDF Files → Text Extraction → JSON Storage
b. User Query → BM25 Search Engine → JSON Files
c. Search Results → Display Components
Specific Implementation Details
• Two main processes:
1. Preprocessing Pipeline (chunk-pdf-pages.py):
PDF Files → PyMuPDF4LLM → Page Chunking → JSON Storage
2. Search Pipeline (BM25-PDF-Search.py):
Query → BM25 → Results → Dual Display (Text + PDF)
Specific elements
• PDF File Repository
• JSON Cache Storage
• PyQt5 GUI Components
• BM25 Search Engine
• Text Extraction Module
• Synchronized Display System
• User Input Handler
• Navigation Controller
Technical Notes
• That it's a file-system based application
• That JSON files mirror PDF files (1:1 relationship)
• Synchronization between text and PDF views
• Search keyword highlighting system
• The simplicity of the architecture (no client-server, no database)
• The two-phase nature of the system (preprocessing and search)
• The dual-view presentation system
• The file-based storage approach
• Data flow directions
• File storage types
• Processing components
• User interface elements
• External dependencies
This will provide a clear understanding of how the system processes PDFs, stores the extracted text, performs searches, and presents results to users.