# oomol-lab/pdf-craft architecture

> Architecture diagram of the GitHub repository oomol-lab/pdf-craft, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oomol-lab/pdf-craft
- Repository: https://github.com/oomol-lab/pdf-craft

## Overview

##### Project Type & Purpose
- PDF Craft is a Python library designed to process scanned PDFs. It converts PDFs into Markdown files or EPUB ebooks by extracting text (using OCR) and analyzing layout/cues from a scanned page.
- The library integrates AI models – including an OCR engine (OnnxOCR), layout detection (DocLayout-YOLO, layoutreader), and LLM services (DeepSeek) – to produce semantically organized output and correct OCR artifacts.
- It is used as a processing pipeline that takes in a PDF, passes it through multiple stages (OCR page extraction, content analysis, correction via LLM) and finally outputs text (Markdown or structured EPUB).

##### Analyzing the File Structure
- There is a clear project package “pdf_craft” with sub-packages:
- "analyser": Contains modules such as asset_matcher, chapter, page, state_machine, etc. These indicate that it’s responsible for examining the PDF’s layout/structure (e.g., splitting pages, handling chapters, citations, positions, etc.).
- "epub": Modules like gen_epub, gen_index, gen_part – these are focused on assembling content into EPUB, managing chapters and the overall ebook structure.
- "llm": Contains modules for integration with LLM services to analyze and improve OCR output (error.py, executor.py, node.py). The LLM component is intended to refine the text, generate structured information, and adapt the document based on LLM responses.
- "pdf": Focuses on lower-level PDF operations (document parsing, text matching, sections) – essentially doing the heavy lifting for extracting text blocks from PDF files.
- Other files like markdown.py and template.py indicate the creation of Markdown output and templating for various target formats.
- The “data” directory holds Jinja templates and prompt files that enable customization and formatting for both analysis and EPUB generation. The presence of both “prompts” and “templates” means that the design is modular with configurable output formats.
- The scripts folder contains sample command-line scripts that invoke various functions (analyzing, generating epub/markdown) to demonstrate how the library should be used.
- Testing modules demonstrate that components such as page clipping and serial extraction have independent tests, which suggests a decoupled design.

##### Key Components & Their Relationships
a. Input/Processing:
– PDF Input: The system takes a PDF file as input.
– PDF Extractor (located in pdf_craft/pdf and PDFPageExtractor referenced in README): Reads pages and extracts initial text and images.
– OCR Module: Uses OnnxOCR (and optionally GPU support via CUDA) to recognize text from scanned images.
b. Analysis:
– The “analyser” package processes extracted content. It handles tasks such as separating header/footer, matching citations, analyzing page layout, linking content between pages, and extracting chapters.
– It uses prompt templates (in the “data/prompts” directory) to interact with the LLM.
c. LLM Integration:
– The “llm” package acts as an interface between the local extraction/analysis and external LLM services (DeepSeek) for error correction and chapter structuring.
– It also handles advanced configuration (retries, temperature control).
d. Output Generators:
– Markdown Writer: Exports the final content to Markdown (pdf_craft/markdown.py).
– EPUB Generator: The “epub” package modules generate EPUB files using the analyzed data and templating system.
– Both generators reference the Jinja templates (in “data/templates”) to consistently format the output.
e. Auxiliary/Supporting Components:
– Scripts provide various usage examples.
– Testing modules ensure quality and proper isolation of functional components.
– CI/CD configuration files and manifests (.github/workflows, setup.py, etc.) help with building, linting, and deployment.

##### Architectural Patterns & Design Principles
- Modularity: Each major function – PDF extraction, analysis, LLM correction, final content generation – is split into its own set of modules and packages.
- Pipeline Architecture: The data flows from input, through extraction, analysis, correction, and finally into one of the output pathways (Markdown or EPUB).
- Template Driven: Use of Jinja templates for both prompts and output formatting is a key design decision.
- Extensibility: The design separates OCR, LLM integration, and output generation, which makes it easier to replace or upgrade individual parts (e.g., switching OCR engines or LLM services).
- Testability: The clear separation into packages and independent test modules (in “tests”) ensures each component can be developed and verified in isolation.

##### Final Advice
- Detailing all essential actor modules – don’t hesitate to break down each high-level component into individual functions where necessary.
- To incorporate configuration and error management flows which are critical in the LLM and OCR modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        PDFInput["PDF Input"]:::input
    end

    %% Extraction Layer
    subgraph "Extraction Layer"
        PDFExtractor["PDF Extractor"]:::extraction
        OCRModule["OCR Module"]:::extraction
    end

    %% Analysis Layer - Direct Path
    subgraph "Analysis Layer (Direct)"
        ContentAnalyzer["Content Analyzer"]:::analysis
        MarkdownWriter["Markdown Writer"]:::analysis
    end

    %% LLM Integration Layer
    subgraph "LLM Integration Layer"
        LLMIntegration["LLM Integration"]:::llm
        LLMContentAnalyzer["Content Analyzer (Post-LLM)"]:::llm
        EPUBGenerator["EPUB Generator"]:::llm
    end

    %% Templating Systems
    subgraph "Templating Systems"
        PromptTemplates["Prompt Templates"]:::templating
        OutputTemplates["Output Templates"]:::templating
    end

    %% Auxiliary Components
    subgraph "Auxiliary"
        CLIScripts["CLI Scripts"]:::auxiliary
    end

    %% Connections between layers
    PDFInput -->|"feeds"| PDFExtractor
    PDFExtractor -->|"parses"| OCRModule
    OCRModule -->|"direct_path"| ContentAnalyzer
    OCRModule -->|"llm_path"| LLMIntegration

    ContentAnalyzer -->|"assembles"| MarkdownWriter
    LLMIntegration -->|"corrects"| LLMContentAnalyzer
    LLMContentAnalyzer -->|"structures"| EPUBGenerator

    %% Templating relationships
    PromptTemplates -->|"provides"| LLMIntegration
    OutputTemplates -->|"formats"| MarkdownWriter
    OutputTemplates -->|"formats"| EPUBGenerator

    %% CLI Scripts trigger the process
    CLIScripts -->|"triggers"| PDFInput

    %% Click Events
    click PDFExtractor "https://github.com/oomol-lab/pdf-craft/blob/main/pdf_craft/pdf/extractor.py"
    click OCRModule "https://github.com/oomol-lab/pdf-craft/blob/main/pdf_craft/analyser/ocr_extractor.py"
    click ContentAnalyzer "https://github.com/oomol-lab/pdf-craft/tree/main/pdf_craft/analyser/"
    click LLMIntegration "https://github.com/oomol-lab/pdf-craft/tree/main/pdf_craft/llm/"
    click MarkdownWriter "https://github.com/oomol-lab/pdf-craft/blob/main/pdf_craft/markdown.py"
    click EPUBGenerator "https://github.com/oomol-lab/pdf-craft/tree/main/pdf_craft/epub/"
    click PromptTemplates "https://github.com/oomol-lab/pdf-craft/tree/main/pdf_craft/data/prompts/"
    click OutputTemplates "https://github.com/oomol-lab/pdf-craft/tree/main/pdf_craft/data/templates/"
    click CLIScripts "https://github.com/oomol-lab/pdf-craft/tree/main/scripts/"

    %% Styles
    classDef input fill:#f9f,stroke:#333,stroke-width:2px;
    classDef extraction fill:#ccf,stroke:#333,stroke-width:2px;
    classDef analysis fill:#cfc,stroke:#333,stroke-width:2px;
    classDef llm fill:#fc9,stroke:#333,stroke-width:2px;
    classDef templating fill:#cff,stroke:#333,stroke-width:2px;
    classDef auxiliary fill:#ffc,stroke:#333,stroke-width:2px;
```
