How luciuswoo/pdf-craft works
Main Architectural Components
• PDF Input Module – This represents the entry point where PDF files (including scanned documents) are provided.
• PDF Extraction Components:
• PDFPageExtractor: Responsible for reading PDF pages one by one.
• OCR / Layout Handling: Although encapsulated inside the extractor, it relies on external OCR and layout recognition tools (e.g., OnnxOCR, DocLayout-YOLO, layoutreader).
• Analysis & Text Processing:
• Analyser module: Located in pdf_craft/analyser, this module further processes the extracted text (segmenting main text, citations, chapters, etc.), handles page segmentation, and corrects artefacts like headers/footers.
• Utility modules and supporting components (e.g., asset_matcher, chunk_file, serial, state_machine) which ensure robust parsing and data integrity.
• LLM Integration:
• LLM module (under pdf_craft/llm): Handles interaction with a remote LLM service (such as DeepSeek) to process OCR data, correct OCR mistakes, analyze annotations, and build the book structure.
• Configuration aspects (top_p, temperature, timeout, and retry logic) that fine-tune the LLM requests.
• Output Generation:
• Markdown Writer (and associated modules) for converting processed text into a Markdown file with linked assets.
• EPUB Generation: Modules under pdf_craft/epub (gen_epub, gen_index, gen_part, etc.) that assemble the analyzed content into a structured EPUB file complete with chapters, TOC, and citations.
• Template & Prompt Management:
• The data/templates and data/prompts directories store Jinja templates and prompts that guide both the extraction and LLM-based analysis phases.
• Supporting Artifacts and Tools:
• Scripts: Containing standalone scripts for tasks such as analyzing PDFs, generating outputs, and token counting.
• Tests: Unit tests validate various subcomponents (e.g., OCR extraction, page clipping, and LLM escaping).
Relationships and Data Flow
• The extracted information is then sent to the analyser modules where it is parsed and segmented. At this stage, designated segments (e.g., main text, citation, metadata) are produced.
• For operations that require language understanding or correction (such as EPUB generation), the segmented content is forwarded to the LLM module. The LLM service (an external component) refines the text, ensures continuity across page breaks, and builds a coherent narrative structure.
• Finally, the processed and refined outputs are passed to the output generators: either converted into Markdown (using the MarkDownWriter) or packaged into an EPUB file using the epub modules.
• The templating system provides the formats and layout for the outputs, pulling in pre-defined templates and prompts from the data directory.
Architectural Patterns and Design Principles
• Modularity: Observe clear functional separation (extraction, analysis, LLM processing, and output generation) so that each module can be understood and maintained independently.
• Pipeline Design: The overall flow resembles a pipeline where data is progressively transformed from raw PDF pages to final formatted documents.
• Extensibility: With external configurations (for LLM service and OCR settings) and a templated approach for outputs, the system is flexible and can be extended or swapped out (for instance, using another OCR engine or LLM).
• Error Handling and Recoverability: Note the handling of interruptions via the “analysing_dir_path” which stores intermediate states.