How polylvst/bca-pdfparser-to-json-to-text-to-db works
Project type and purpose
• This is an open-source, command-line PDF parsing tool (“PDF Parser BCA”). It ingests PDF statements, extracts structured data, and emits it to one of three sinks: a relational database, JSON files, or plain-text files.
File structure
• Top-level scripts:
• run.py → orchestrates end-to-end import into the database
• tokenize_util.py → tokenizes PDF pages and writes JSON
• parse.py → runs text extraction, writes .txt output
• calculate_balance.py, prepare_dict.py → helper modules for data derivation
• db.py → database connection and persistence logic
• config.py + config.yaml → configuration loader and settings
• shared_enum.py → common enums used across scripts
• utils/area_picker.py: shared utility for selecting PDF regions
• requirements.txt: lists Python dependencies
README for additional insights
• Workflow steps:
a. Drop PDFs in an input folder
b. Edit config.yaml (DB credentials, output paths)
c. Run run.py for DB import
d. Run tokenize_util.py for JSON export
e. Run parse.py for TXT export
Main components and their interactions
a. Input Component
• “Input Folder” (local file system) containing PDF files
b. Configuration Component
• config.yaml loaded by config.py → central settings object
c. Parsing Pipeline
1. PDF Reader & Tokenizer (tokenize_util.py + utils/area_picker.py)
• Reads PDF pages
• Splits into tokens/regions
• Emits JSON per page or document
2. Dictionary Preparer (prepare_dict.py) & Balance Calculator (calculate_balance.py)
• Builds domain dictionaries (e.g., transaction codes, categories)
• Computes running balances or aggregates
3. Text Parser (parse.py)
• Applies region-based heuristics → outputs plain text transcripts
d. Persistence / Output
• Database Module (db.py)
• Receives structured records → writes to relational DB (via ORM or raw SQL)
• JSON Module (tokenize_util.py) output
• Text Module (parse.py) output
Architectural patterns & design principles
• Pipeline pattern: a series of transform steps (tokenize → prepare → calculate → persist)
• Single-responsibility modules: each script has one clear output target (DB, JSON, TXT)
• Configuration centralization: config.yaml + config.py for all environment settings
• Reusable utilities: shared_enum.py and utils/area_picker.py used by multiple scripts