How hakanotal/pdf2kg works
Main Components
• Input Data (PDF files): This is where the process begins. Represent the raw PDFs that feed into the pipeline.
• Filter Component: Show the "Filter Small PDFs" module (0_filter_small_pdfs.py) that screens out PDFs below a size threshold.
• Conversion Component: Represent the conversion from PDF to Markdown (1_pdf_to_md.sh). This module converts the PDFs filtered in the previous step into Markdown format.
• Knowledge Graph Generation: Illustrate the "Markdown to KG" component (2_md_to_kg.py) where the Markdown files are processed to extract entities, relationships, and structure the information into a knowledge graph.
• Supporting Utilities: Include the "utils" package (with its helpers and prompts modules), which provides common functions and shared logic used across multiple pipeline steps.
Relationships and Interactions
• A clear sequential flow: Start with raw PDF files, then pass through filtering, conversion, knowledge graph extraction, and finishing with post-processing.
• Interactions to show key inputs and outputs—mention, for example, that the Filter outputs a subset of PDFs that are then used by the Conversion module, which outputs Markdown files, and so on.
Architectural Patterns
• Extensibility: Optionally, note that additional modules could be added or replaced in the pipeline in future development.
Relevant Technologies and Tools
• Scripting Languages: Mention that Python is used for processing scripts and Shell for conversion.
• External Tools: Briefly note the dependency on external PDF processing tools (as referenced in the README).
• Environment: Indicate the need for Python 3.11 and dependencies installed via requirements.txt.