How iamarunbrahma/pdf-to-markdown works
Project Overview and Main Components
• This project is a utility tool designed to convert PDF files into markdown-formatted output, ensuring preservation of formatting elements (e.g., tables, images, bold/italic text, code blocks) while also employing OCR for images.
• Following key components:
• Input Module: Handles receiving or specifying the PDF file (via command-line arguments).
• Configuration Manager: Reads settings from the config file (config/config.yaml) to configure extraction parameters.
• Extraction Engine: The central module in “extract.py” that orchestrates the extraction process.
• PDF Processing Submodules: Uses libraries like PyMuPDF (fitz) and pdfplumber to parse and extract text and layout.
• OCR and Image Processing Submodule: Utilizes pytesseract and OpenCV (cv2) for processing images and performing OCR; integrates with transformer-based models for image caption generation.
• Markdown Conversion Module: Takes the parsed data and converts it into well-structured Markdown output.
• Output Writer: Saves the resulting .md file into the intended output directory.
• Dependency Layers: Integration with external libraries such as numpy, Pillow, transformers, and torch.
Key Relationships and Interactions
• The Input Module feeds the PDF file to the Extraction Engine.
• The Extraction Engine interacts with:
• The Configuration Manager for settings.
• The PDF Parsing and OCR submodules in parallel.
• Data flows from the parsing modules (text extraction and OCR) to the Markdown Conversion module.
• The Markdown Conversion module then pushes the final formatted markdown to the Output Writer.
Architectural Patterns and Design Principles
• Separation of concerns: Different modules handle specific tasks (e.g., PDF parsing vs. OCR).
• Modular design: Each processing step (parsing, OCR, conversion) is encapsulated, making future extensions or debugging easier.
• Dependency injection: The configuration management component provides parameters across modules.
• Adaptability: The design supports handling both native text PDFs and scanned images.
Relevant Technologies and Libraries
• Python is the primary language.
• PyMuPDF (fitz) and pdfplumber for reading PDF content.
• pytesseract and OpenCV (cv2) for OCR and image processing.
• Additional libraries like numpy, Pillow, transformers, and torch assist in handling complex processing like OCR enhancements and image captioning.
• Configuration management via YAML (config.yaml).
Specific Elements
• Clear labels for each module/component.
• PDF Input Module
• Configuration Manager (config/config.yaml)
• Extraction Engine split into:
– PDF Parsing (PyMuPDF, pdfplumber)
– OCR & Image Processing (pytesseract, OpenCV, image caption generation)
• Markdown Conversion Module
• Output Writer (producing the .md file)