# polylvst/bca-pdfparser-to-json-to-text-to-db architecture

> Architecture diagram of the GitHub repository polylvst/bca-pdfparser-to-json-to-text-to-db, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/polylvst/bca-pdfparser-to-json-to-text-to-db
- Repository: https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db

## Overview

##### Project type and purpose
- This is an open-source, command-line PDF parsing tool (“PDF Parser BCA”). It ingests PDF statements, extracts structured data, and emits it to one of three sinks: a relational database, JSON files, or plain-text files.

##### File structure
- Top-level scripts:
- run.py        → orchestrates end-to-end import into the database
- tokenize_util.py → tokenizes PDF pages and writes JSON
- parse.py      → runs text extraction, writes .txt output
- calculate_balance.py, prepare_dict.py → helper modules for data derivation
- db.py         → database connection and persistence logic
- config.py + config.yaml → configuration loader and settings
- shared_enum.py → common enums used across scripts
- utils/area_picker.py: shared utility for selecting PDF regions
- requirements.txt: lists Python dependencies

##### README for additional insights
- Workflow steps:
a. Drop PDFs in an input folder
b. Edit config.yaml (DB credentials, output paths)
c. Run run.py for DB import
d. Run tokenize_util.py for JSON export
e. Run parse.py for TXT export

##### Main components and their interactions
a. Input Component
- “Input Folder” (local file system) containing PDF files
b. Configuration Component
- config.yaml loaded by config.py → central settings object
c. Parsing Pipeline
1. PDF Reader & Tokenizer (tokenize_util.py + utils/area_picker.py)
- Reads PDF pages
- Splits into tokens/regions
- Emits JSON per page or document
2. Dictionary Preparer (prepare_dict.py) & Balance Calculator (calculate_balance.py)
- Builds domain dictionaries (e.g., transaction codes, categories)
- Computes running balances or aggregates
3. Text Parser (parse.py)
- Applies region-based heuristics → outputs plain text transcripts
d. Persistence / Output
- Database Module (db.py)
- Receives structured records → writes to relational DB (via ORM or raw SQL)
- JSON Module (tokenize_util.py) output
- Text Module (parse.py) output

##### Architectural patterns & design principles
- Pipeline pattern: a series of transform steps (tokenize → prepare → calculate → persist)
- Single-responsibility modules: each script has one clear output target (DB, JSON, TXT)
- Configuration centralization: config.yaml + config.py for all environment settings
- Reusable utilities: shared_enum.py and utils/area_picker.py used by multiple scripts

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    InputFolder["Input Folder"]:::dataSource
    ConfigLoader["Config Loader\n(config.py & config.yaml)"]:::dataSource

    %% Parsing Pipeline
    subgraph "Parsing Pipeline"
        PDFTokenizer["PDF Tokenizer\n(tokenize_util.py)"]:::processing
        DictPreparer["Dictionary Preparer\n(prepare_dict.py)"]:::processing
        BalanceCalculator["Balance Calculator\n(calculate_balance.py)"]:::processing
        TextParser["Text Parser\n(parse.py)"]:::processing
    end

    %% Utilities
    RegionPicker["Region-Picker Utility\n(utils/area_picker.py)"]:::util
    SharedEnums["Shared Enums\n(shared_enum.py)"]:::util

    %% Outputs
    JSONOutput["JSON Output\n(.json files)"]:::output
    TXTOutput["TXT Output\n(.txt files)"]:::output
    DBPersister["DB Persister\n(db.py & run.py)"]:::output

    %% Data Flows
    InputFolder --> PDFTokenizer
    ConfigLoader --> PDFTokenizer
    ConfigLoader --> DictPreparer
    ConfigLoader --> BalanceCalculator
    ConfigLoader --> TextParser

    PDFTokenizer --> JSONOutput
    PDFTokenizer --> DictPreparer
    PDFTokenizer --> TextParser

    DictPreparer --> BalanceCalculator
    BalanceCalculator --> DBPersister
    TextParser --> TXTOutput

    RegionPicker --> PDFTokenizer
    SharedEnums --> PDFTokenizer
    SharedEnums --> DictPreparer
    SharedEnums --> BalanceCalculator
    SharedEnums --> TextParser
    SharedEnums --> DBPersister

    %% Click Events
    click ConfigLoader "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/config.py"
    click PDFTokenizer "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/tokenize_util.py"
    click RegionPicker "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/utils/area_picker.py"
    click DictPreparer "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/prepare_dict.py"
    click BalanceCalculator "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/calculate_balance.py"
    click TextParser "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/parse.py"
    click DBPersister "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/db.py"
    click SharedEnums "https://github.com/polylvst/bca-pdfparser-to-json-to-text-to-db/blob/main/shared_enum.py"

    %% Styles
    classDef dataSource fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef processing fill:#90EE90,stroke:#333,stroke-width:1px
    classDef output fill:#FFA500,stroke:#333,stroke-width:1px
    classDef util stroke-dasharray: 5 5,stroke:#666,fill:#fff,stroke-width:1px
```
