How callahantiff/pheknowlator works
• Data Input/Resource Layer:
• The resources/ directory holds all input dependency files (ontology_source_list.txt, edge_source_list.txt, resource_info.txt, and additional documents in subdirectories such as construction_approach, edge_data, ontologies, etc.). This is the foundation for downloading and pre-processing data.
• The notebooks/ directory (with Data_Preparation.ipynb and tutorials) shows how users can prepare or inspect input data.
• Data Download and Preprocessing:
• The “downloads.py” module and associated functions in pkt_kg handle fetching data from URLs (both ontologies and edge sources).
• The preprocessing scripts (e.g., data_preprocessing.py in the builds/ directory) and cleaning scripts (ontology_cleaning.py) set up the input dependencies.
• Master Edge List Construction:
• Use the edge_list.py and related modules that create and merge edge data into a “master edge list.” This is critical as it combines data from various sources.
• Knowledge Graph Construction Engine:
• The core logic in modules like knowledge_graph.py, owlnets.py, and construction_approaches.py define how the KG is built. This part also handles options such as including inverse relations, node metadata, and decoding OWL semantics.
• The “main.py” script and “main.ipynb” notebook act as orchestrators for this process.
• Build Pipeline/Scripts:
• The builds/ folder contains multiple phases (e.g., build_phase_1.py, build_phase_2.py, build_phase_3.py) and deployment scripts (including the docker-compose configuration under builds/deploy/triple-store) that show a multi-stage build process.
• Execution Environment & Orchestration:
• Docker support (Dockerfile at the root and in builds) indicates that part of the system design is containerized.
• GitHub Actions and CI configuration files (within .github/workflows) indicate continuous integration and perhaps automated testing/deployment pipelines.
• Testing and Monitoring:
• The tests/ directory holds regression and functionality tests for each component to ensure robustness.
• Job monitoring scripts (job_monitoring.py) are also part of the pipeline.
Relationships and Data Flow
• Data flows initially from the static resources (or via downloads) into the “Download Data” modules.
• Once downloaded, the data are processed and metadata files are created (e.g., ontology_source_metadata.txt and edge_source_metadata.txt).
• The processed data are merged to create a master edge list, serving as input to the core Knowledge Graph Construction Engine.
• The KG Construction Engine (possibly executed via main.py, controlled via command-line options or through the Jupyter Notebook) creates the final output files in the resources/knowledge_graphs directory.
• Optionally, if running in Docker, the container abstracts these internal steps, mounting volumes to persist output data.
• Parallel processing via Ray is initialized from the main orchestrator to distribute heavy computational tasks like edge list construction and graph assembly.
Architectural Patterns and Principles
• Modular design: Separation of concerns among data download, pre-processing, construction, and test modules.
• Pipeline pattern: A sequential data processing pipeline from raw data download to construction and final graph output.
• Extensibility: Emphasis on customization options (e.g., different construction approaches, support for inverse relations, node metadata, and OWL decoding) signals a design that can be easily extended.
• Containerization & CI/CD: Docker support and GitHub workflows hint at a modern deployment and testing strategy.