How oomol-lab/pdf-craft works
Project Type & Purpose
• PDF Craft is a Python library designed to process scanned PDFs. It converts PDFs into Markdown files or EPUB ebooks by extracting text (using OCR) and analyzing layout/cues from a scanned page.
• The library integrates AI models – including an OCR engine (OnnxOCR), layout detection (DocLayout-YOLO, layoutreader), and LLM services (DeepSeek) – to produce semantically organized output and correct OCR artifacts.
• It is used as a processing pipeline that takes in a PDF, passes it through multiple stages (OCR page extraction, content analysis, correction via LLM) and finally outputs text (Markdown or structured EPUB).
Analyzing the File Structure
• There is a clear project package “pdf_craft” with sub-packages:
• "analyser": Contains modules such as asset_matcher, chapter, page, state_machine, etc. These indicate that it’s responsible for examining the PDF’s layout/structure (e.g., splitting pages, handling chapters, citations, positions, etc.).
• "epub": Modules like gen_epub, gen_index, gen_part – these are focused on assembling content into EPUB, managing chapters and the overall ebook structure.
• "llm": Contains modules for integration with LLM services to analyze and improve OCR output (error.py, executor.py, node.py). The LLM component is intended to refine the text, generate structured information, and adapt the document based on LLM responses.
• "pdf": Focuses on lower-level PDF operations (document parsing, text matching, sections) – essentially doing the heavy lifting for extracting text blocks from PDF files.
• Other files like markdown.py and template.py indicate the creation of Markdown output and templating for various target formats.
• The “data” directory holds Jinja templates and prompt files that enable customization and formatting for both analysis and EPUB generation. The presence of both “prompts” and “templates” means that the design is modular with configurable output formats.
• The scripts folder contains sample command-line scripts that invoke various functions (analyzing, generating epub/markdown) to demonstrate how the library should be used.
• Testing modules demonstrate that components such as page clipping and serial extraction have independent tests, which suggests a decoupled design.
Key Components & Their Relationships
a. Input/Processing:
– PDF Input: The system takes a PDF file as input.
– PDF Extractor (located in pdf_craft/pdf and PDFPageExtractor referenced in README): Reads pages and extracts initial text and images.
– OCR Module: Uses OnnxOCR (and optionally GPU support via CUDA) to recognize text from scanned images.
b. Analysis:
– The “analyser” package processes extracted content. It handles tasks such as separating header/footer, matching citations, analyzing page layout, linking content between pages, and extracting chapters.
– It uses prompt templates (in the “data/prompts” directory) to interact with the LLM.
c. LLM Integration:
– The “llm” package acts as an interface between the local extraction/analysis and external LLM services (DeepSeek) for error correction and chapter structuring.
– It also handles advanced configuration (retries, temperature control).
d. Output Generators:
– Markdown Writer: Exports the final content to Markdown (pdf_craft/markdown.py).
– EPUB Generator: The “epub” package modules generate EPUB files using the analyzed data and templating system.
– Both generators reference the Jinja templates (in “data/templates”) to consistently format the output.
e. Auxiliary/Supporting Components:
– Scripts provide various usage examples.
– Testing modules ensure quality and proper isolation of functional components.
– CI/CD configuration files and manifests (.github/workflows, setup.py, etc.) help with building, linting, and deployment.
Architectural Patterns & Design Principles
• Modularity: Each major function – PDF extraction, analysis, LLM correction, final content generation – is split into its own set of modules and packages.
• Pipeline Architecture: The data flows from input, through extraction, analysis, correction, and finally into one of the output pathways (Markdown or EPUB).
• Template Driven: Use of Jinja templates for both prompts and output formatting is a key design decision.
• Extensibility: The design separates OCR, LLM integration, and output generation, which makes it easier to replace or upgrade individual parts (e.g., switching OCR engines or LLM services).
• Testability: The clear separation into packages and independent test modules (in “tests”) ensures each component can be developed and verified in isolation.
Final Advice
• Detailing all essential actor modules – don’t hesitate to break down each high-level component into individual functions where necessary.
• To incorporate configuration and error management flows which are critical in the LLM and OCR modules.