How muhammad-azmain-mahtab/automated-pdf-to-excel works
Project Type and Purpose
• This is a specialized data‐processing pipeline for converting table PDFs from financial reports into Excel/HTML formats and generating annotation ground‐truth in JSONL format.
• It combines a series of Python GUI scripts and external executables (PPOCRLabelv2, Data_Processing.exe) to guide a user through PDF‐to‐image conversion, table cropping, annotation, HTML generation, and final JSONL creation.
File Structure
Top-level scripts represent discrete pipeline stages:
• pdf_to_image_gui.py – GUI for converting PDFs into images
• extract_html_stucture_gui.py – GUI for extracting table HTML structures from Excel
• excel_to_html_stucture_to_gt_jsonl_gui.py – GUI for converting Excel→HTML→gt.jsonl
• Merge_all_image_and_gt.py, Merge_image_and_gt_in_subfolder.py – scripts to merge images with annotations
• train_test_split.py – splits final dataset
• main.py – (optional) orchestrates or launches the above
No nested directories—each script is a self‐contained component.
Extract Insights from README
• External dependencies: Anaconda environment, paddlepaddle, PPOCRLabel, pywin32.
• Two auxiliary executables: pdf_to_image_gui.exe for image conversion; Data_Processing.exe for Excel→HTML and gt.jsonl generation.
a. Main Components
• User Workstation (Windows + Anaconda env)
• Python GUI Modules (pdf_to_image_gui, extract_html_structure_gui, excel_to_html_structure_to_gt_jsonl_gui)
• External Annotation Tool (PPOCRLabelv2)
• External Converter (Data_Processing.exe)
• File System Storage (folders for PDFs, images, Excel files, HTML files, annotation outputs, final JSONL)
• Merge & Validation Scripts
• Final Dataset (gt.jsonl)
b. Relationships & Data Flow
1. User drops PDF folder → pdf_to_image_gui → Converts PDFs → Stores images
2. User crops table images manually → Images folder → PPOCRLabelv2 → Generates gt.txt and updated Excel
3. Data_Processing.exe → Reads Excel files → Outputs HTML structures
4. extract_html_structure_gui → (if separate) assists user in verifying HTML
5. excel_to_html_structure_to_gt_jsonl_gui → Reads HTML + gt.txt → Produces gt.jsonl
7. Merge scripts combine image paths + JSONL into single dataset
8. train_test_split.py splits dataset into train/test sets
c. Architectural Patterns & Principles
• Linear Pipeline: each script is a stage with clear input & output
• Separation of Concerns: GUIs orchestrate I/O; processing logic encapsulated in helper scripts; manual annotation isolated in PPOCRLabel
• Extensibility Points: new converters or pre‐processing scripts can plug in between stages
d. Key Technologies
• Python 3.10, pandas, OpenCV/Tesseract (implied), PyWin32 for GUI
• PaddlePaddle for PPOCRLabelv2’s table recognition model
• External executables: Data_Processing.exe, pdf_to_image_gui.exe