# muhammad-azmain-mahtab/automated-pdf-to-excel architecture

> Architecture diagram of the GitHub repository muhammad-azmain-mahtab/automated-pdf-to-excel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/muhammad-azmain-mahtab/automated-pdf-to-excel
- Repository: https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel

## Overview

##### Project Type and Purpose
- This is a specialized data‐processing pipeline for converting table PDFs from financial reports into Excel/HTML formats and generating annotation ground‐truth in JSONL format.
- It combines a series of Python GUI scripts and external executables (PPOCRLabelv2, Data_Processing.exe) to guide a user through PDF‐to‐image conversion, table cropping, annotation, HTML generation, and final JSONL creation.

##### File Structure
Top-level scripts represent discrete pipeline stages:
- pdf_to_image_gui.py – GUI for converting PDFs into images
- extract_html_stucture_gui.py – GUI for extracting table HTML structures from Excel
- excel_to_html_stucture_to_gt_jsonl_gui.py – GUI for converting Excel→HTML→gt.jsonl
- Merge_all_image_and_gt.py, Merge_image_and_gt_in_subfolder.py – scripts to merge images with annotations
- train_test_split.py – splits final dataset
- main.py – (optional) orchestrates or launches the above
No nested directories—each script is a self‐contained component.

##### Extract Insights from README
- External dependencies: Anaconda environment, paddlepaddle, PPOCRLabel, pywin32.
- Two auxiliary executables: pdf_to_image_gui.exe for image conversion; Data_Processing.exe for Excel→HTML and gt.jsonl generation.

a. Main Components
- User Workstation (Windows + Anaconda env)
- Python GUI Modules (pdf_to_image_gui, extract_html_structure_gui, excel_to_html_structure_to_gt_jsonl_gui)
- External Annotation Tool (PPOCRLabelv2)
- External Converter (Data_Processing.exe)
- File System Storage (folders for PDFs, images, Excel files, HTML files, annotation outputs, final JSONL)
- Merge & Validation Scripts
- Final Dataset (gt.jsonl)
b. Relationships & Data Flow
1. User drops PDF folder → pdf_to_image_gui → Converts PDFs → Stores images
2. User crops table images manually → Images folder → PPOCRLabelv2 → Generates gt.txt and updated Excel
3. Data_Processing.exe → Reads Excel files → Outputs HTML structures
4. extract_html_structure_gui → (if separate) assists user in verifying HTML
5. excel_to_html_structure_to_gt_jsonl_gui → Reads HTML + gt.txt → Produces gt.jsonl
7. Merge scripts combine image paths + JSONL into single dataset
8. train_test_split.py splits dataset into train/test sets
c. Architectural Patterns & Principles
- Linear Pipeline: each script is a stage with clear input & output
- Separation of Concerns: GUIs orchestrate I/O; processing logic encapsulated in helper scripts; manual annotation isolated in PPOCRLabel
- Extensibility Points: new converters or pre‐processing scripts can plug in between stages
d. Key Technologies
- Python 3.10, pandas, OpenCV/Tesseract (implied), PyWin32 for GUI
- PaddlePaddle for PPOCRLabelv2’s table recognition model
- External executables: Data_Processing.exe, pdf_to_image_gui.exe

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Environment Orchestration
    main.py["main.py"]:::python
    main.py -.-> pdf2imgGUI
    main.py -.-> extractHTMLGUI
    main.py -.-> excelToHTMLGUI

    %% Input Stage
    PDFInput["PDF Input Folder"]:::storage
    PDFInput -->|"drop PDFs"| pdf2imgGUI["pdf_to_image_gui.py"]:::python

    %% PDF to Image Conversion
    pdf2imgGUI -->|"calls .exe"| pdfExe["pdf_to_image_gui.exe"]:::exec
    pdfExe -->|"outputs .png"| ImageFolder["Image Folder"]:::storage

    %% Manual Annotation
    ImageFolder -->|"manual crop"| PPOCR["PPOCRLabelv2 Annotation"]:::manual
    PPOCR -->|"outputs gt.txt & .xlsx"| ExcelFolder["Excel Folder"]:::storage

    %% Excel to HTML Conversion
    ExcelFolder -->|"reads .xlsx"| dataProcExe["Data_Processing.exe"]:::exec
    dataProcExe -->|"outputs .html"| HTMLFolder["HTML Folder"]:::storage

    %% HTML Verification GUI
    HTMLFolder --> extractHTMLGUI["extract_html_stucture_gui.py"]:::python
    extractHTMLGUI --> HTMLFolder

    %% HTML + Annotation to JSONL
    HTMLFolder -->|"reads .html + gt.txt"| excelToHTMLGUI["excel_to_html_stucture_to_gt_jsonl_gui.py"]:::python
    excelToHTMLGUI --> JSONLFolder["Ground-Truth JSONL Folder"]:::storage

    %% Validation
    JSONLFolder --> bboxValidate["bbox_cell_count_check.py"]:::python
    bboxValidate --> JSONLFolder

    %% Merging Datasets
    ImageFolder --> mergeAll["Merge_all_image_and_gt.py"]:::python
    JSONLFolder --> mergeAll
    mergeAll --> MergedDataset["Merged Dataset Folder"]:::storage

    ImageFolder --> mergeSub["Merge_image_and_gt_in_subfolder.py"]:::python
    JSONLFolder --> mergeSub
    mergeSub --> MergedDataset

    %% Train/Test Split
    MergedDataset --> splitTrainTest["train_test_split.py"]:::python
    splitTrainTest --> TrainSet["Train Set Folder"]:::storage
    splitTrainTest --> TestSet["Test Set Folder"]:::storage

    %% Click Events
    click pdf2imgGUI "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/pdf_to_image_gui.py"
    click extractHTMLGUI "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/extract_html_stucture_gui.py"
    click excelToHTMLGUI "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/excel_to_html_stucture_to_gt_jsonl_gui.py"
    click mergeAll "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/Merge_all_image_and_gt.py"
    click mergeSub "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/Merge_image_and_gt_in_subfolder.py"
    click bboxValidate "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/bbox_cell_count_check.py"
    click splitTrainTest "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/train_test_split.py"
    click main.py "https://github.com/muhammad-azmain-mahtab/automated-pdf-to-excel/blob/main/main.py"

    %% Styles
    classDef python fill:#daeaff,stroke:#1f78b4,stroke-width:1px;
    classDef exec fill:#e0f7e9,stroke:#33a02c,stroke-width:1px;
    classDef storage fill:#ffe5b4,stroke:#ff7f00,stroke-width:1px;
    classDef manual fill:#f0f0f0,stroke:#6c6c6c,stroke-width:1px,stroke-dasharray: 3 3
```
