# hakanotal/pdf2kg architecture

> Architecture diagram of the GitHub repository hakanotal/pdf2kg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hakanotal/pdf2kg
- Repository: https://github.com/hakanotal/pdf2kg

## Overview

##### Main Components
- Input Data (PDF files): This is where the process begins. Represent the raw PDFs that feed into the pipeline.
- Filter Component: Show the "Filter Small PDFs" module (0_filter_small_pdfs.py) that screens out PDFs below a size threshold.
- Conversion Component: Represent the conversion from PDF to Markdown (1_pdf_to_md.sh). This module converts the PDFs filtered in the previous step into Markdown format.
- Knowledge Graph Generation: Illustrate the "Markdown to KG" component (2_md_to_kg.py) where the Markdown files are processed to extract entities, relationships, and structure the information into a knowledge graph.
- Supporting Utilities: Include the "utils" package (with its helpers and prompts modules), which provides common functions and shared logic used across multiple pipeline steps.

##### Relationships and Interactions
- A clear sequential flow: Start with raw PDF files, then pass through filtering, conversion, knowledge graph extraction, and finishing with post-processing.
- Interactions to show key inputs and outputs—mention, for example, that the Filter outputs a subset of PDFs that are then used by the Conversion module, which outputs Markdown files, and so on.

##### Architectural Patterns
- Extensibility: Optionally, note that additional modules could be added or replaced in the pipeline in future development.

##### Relevant Technologies and Tools
- Scripting Languages: Mention that Python is used for processing scripts and Shell for conversion.
- External Tools: Briefly note the dependency on external PDF processing tools (as referenced in the README).
- Environment: Indicate the need for Python 3.11 and dependencies installed via requirements.txt.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Input Data (PDF files)"]:::input
    G["Pipeline Orchestration (run.sh)"]:::orchestration
    B["Filter - 0_filter_small_pdfs.py"]:::process
    C["Conversion - 1_pdf_to_md.sh"]:::process
    D["KG Generation - 2_md_to_kg.py"]:::process
    E["KG Post-Processing - 3_postprocess_kg.py"]:::process
    F["Supporting Utilities (utils)"]:::utils

    %% Pipeline Flow
    G --> A
    A --> B
    B --> C
    C --> D
    D --> E

    %% Supporting Utilities used by processing components
    B -.->|"uses"| F
    C -.->|"uses"| F
    D -.->|"uses"| F
    E -.->|"uses"| F

    %% Click Events
    click B "https://github.com/hakanotal/pdf2kg/blob/main/0_filter_small_pdfs.py"
    click C "https://github.com/hakanotal/pdf2kg/blob/main/1_pdf_to_md.sh"
    click D "https://github.com/hakanotal/pdf2kg/blob/main/2_md_to_kg.py"
    click E "https://github.com/hakanotal/pdf2kg/blob/main/3_postprocess_kg.py"
    click F "https://github.com/hakanotal/pdf2kg/tree/main/utils/"
    click G "https://github.com/hakanotal/pdf2kg/blob/main/run.sh"

    %% Styles
    classDef input fill:#a1c4fd,stroke:#333,stroke-width:2px;
    classDef process fill:#fdd,stroke:#333,stroke-width:2px;
    classDef utils fill:#d4edda,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#fff3cd,stroke:#333,stroke-width:2px;
```
