# callahantiff/pheknowlator architecture

> Architecture diagram of the GitHub repository callahantiff/pheknowlator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/callahantiff/pheknowlator
- Repository: https://github.com/callahantiff/pheknowlator

## Overview

- Data Input/Resource Layer:
- The resources/ directory holds all input dependency files (ontology_source_list.txt, edge_source_list.txt, resource_info.txt, and additional documents in subdirectories such as construction_approach, edge_data, ontologies, etc.). This is the foundation for downloading and pre-processing data.
- The notebooks/ directory (with Data_Preparation.ipynb and tutorials) shows how users can prepare or inspect input data.
- Data Download and Preprocessing:
- The “downloads.py” module and associated functions in pkt_kg handle fetching data from URLs (both ontologies and edge sources).
- The preprocessing scripts (e.g., data_preprocessing.py in the builds/ directory) and cleaning scripts (ontology_cleaning.py) set up the input dependencies.
- Master Edge List Construction:
- Use the edge_list.py and related modules that create and merge edge data into a “master edge list.” This is critical as it combines data from various sources.
- Knowledge Graph Construction Engine:
- The core logic in modules like knowledge_graph.py, owlnets.py, and construction_approaches.py define how the KG is built. This part also handles options such as including inverse relations, node metadata, and decoding OWL semantics.
- The “main.py” script and “main.ipynb” notebook act as orchestrators for this process.
- Build Pipeline/Scripts:
- The builds/ folder contains multiple phases (e.g., build_phase_1.py, build_phase_2.py, build_phase_3.py) and deployment scripts (including the docker-compose configuration under builds/deploy/triple-store) that show a multi-stage build process.
- Execution Environment & Orchestration:
- Docker support (Dockerfile at the root and in builds) indicates that part of the system design is containerized.
- GitHub Actions and CI configuration files (within .github/workflows) indicate continuous integration and perhaps automated testing/deployment pipelines.
- Testing and Monitoring:
- The tests/ directory holds regression and functionality tests for each component to ensure robustness.
- Job monitoring scripts (job_monitoring.py) are also part of the pipeline.

##### Relationships and Data Flow
- Data flows initially from the static resources (or via downloads) into the “Download Data” modules.
- Once downloaded, the data are processed and metadata files are created (e.g., ontology_source_metadata.txt and edge_source_metadata.txt).
- The processed data are merged to create a master edge list, serving as input to the core Knowledge Graph Construction Engine.
- The KG Construction Engine (possibly executed via main.py, controlled via command-line options or through the Jupyter Notebook) creates the final output files in the resources/knowledge_graphs directory.
- Optionally, if running in Docker, the container abstracts these internal steps, mounting volumes to persist output data.
- Parallel processing via Ray is initialized from the main orchestrator to distribute heavy computational tasks like edge list construction and graph assembly.

##### Architectural Patterns and Principles
- Modular design: Separation of concerns among data download, pre-processing, construction, and test modules.
- Pipeline pattern: A sequential data processing pipeline from raw data download to construction and final graph output.
- Extensibility: Emphasis on customization options (e.g., different construction approaches, support for inverse relations, node metadata, and OWL decoding) signals a design that can be easily extended.
- Containerization & CI/CD: Docker support and GitHub workflows hint at a modern deployment and testing strategy.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Resource/Input Layer
    subgraph "Resource/Input Layer"
        R1["Resources"] 
        R2["Notebooks"]
    end

    %% Data Download & Preprocessing Module
    subgraph "Download & Preprocessing Module"
        D1["downloads.py"]
        D2["data_preprocessing.py"]
        D3["ontology_cleaning.py"]
    end

    %% Master Edge List Generator
    subgraph "Master Edge List Generator"
        E1["edge_list.py"]
    end

    %% Knowledge Graph Construction Engine
    subgraph "Knowledge Graph Construction Engine"
        K1["knowledge_graph.py"]
        K2["owlnets.py"]
        K3["construction_approaches.py"]
        K4["Orchestrator (Main.py/main.ipynb)"]
        K5["Ray Parallel Processing"]
    end

    %% Build Pipeline & Orchestration
    subgraph "Build Pipeline & Orchestration"
        B1["Build Scripts (builds/)"]
        B2["Deployment (triple-store)"]
        B3["Dockerfiles"]
    end

    %% Execution Environment & CI/CD
    subgraph "Execution Environment & CI/CD"
        C1["GitHub Actions (.github/workflows)"]
    end

    %% Testing & Monitoring
    subgraph "Testing & Monitoring"
        T1["Tests (tests/)"]
        T2["Job Monitoring"]
    end

    %% Data Flow
    R1 -->|"provides input data"| D1
    D1 -->|"fetches data"| D2
    D2 -->|"prepares data"| D3
    D3 -->|"merges edge lists"| E1
    E1 -->|"supplies master list"| K1
    K1 -->|"applies OWL logic"| K2
    K1 -->|"uses construction methods"| K3
    K1 -->|"orchestrates KG build"| K4
    K4 -->|"distributes tasks"| K5
    R2 -->|"interactive execution"| K4
    K4 -->|"initiates pipeline"| B1
    B1 -->|"deploys container"| B2
    B1 -->|"containerizes build"| B3
    B3 -->|"integrates with"| C1
    C1 -->|"triggers testing"| T1
    C1 -->|"monitors jobs"| T2

    %% Click Events
    click R1 "https://github.com/callahantiff/pheknowlator/tree/master/resources/"
    click R2 "https://github.com/callahantiff/pheknowlator/tree/master/notebooks/"
    click D1 "https://github.com/callahantiff/pheknowlator/blob/master/pkt_kg/downloads.py"
    click D2 "https://github.com/callahantiff/pheknowlator/blob/master/builds/data_preprocessing.py"
    click D3 "https://github.com/callahantiff/pheknowlator/blob/master/builds/ontology_cleaning.py"
    click E1 "https://github.com/callahantiff/pheknowlator/blob/master/pkt_kg/edge_list.py"
    click K1 "https://github.com/callahantiff/pheknowlator/blob/master/pkt_kg/knowledge_graph.py"
    click K2 "https://github.com/callahantiff/pheknowlator/blob/master/pkt_kg/owlnets.py"
    click K3 "https://github.com/callahantiff/pheknowlator/blob/master/pkt_kg/construction_approaches.py"
    click K4 "https://github.com/callahantiff/pheknowlator/blob/master/Main.py"
    click B1 "https://github.com/callahantiff/pheknowlator/tree/master/builds/"
    click B2 "https://github.com/callahantiff/pheknowlator/tree/master/builds/deploy/triple-store"
    click B3 "https://github.com/callahantiff/pheknowlator/tree/master/Dockerfile"
    click C1 "https://github.com/callahantiff/pheknowlator/tree/master/.github/workflows/"
    click T1 "https://github.com/callahantiff/pheknowlator/tree/master/tests/"
    click T2 "https://github.com/callahantiff/pheknowlator/blob/master/builds/job_monitoring.py"

    %% Styles
    class R1,R2 input;
    class D1,D2,D3,E1,K1,K2,K3,K4,K5 process;
    class B1,B2,B3,C1 build;
    class T1,T2 test;

    classDef input fill:#add8e6,stroke:#000,stroke-width:1px;
    classDef process fill:#90ee90,stroke:#000,stroke-width:1px;
    classDef build fill:#ffa07a,stroke:#000,stroke-width:1px;
    classDef test fill:#d3d3d3,stroke:#000,stroke-width:1px;
```
