How rahulnyk/knowledge_graph works
• Data Input & Ingestion Module:
• Directories in “data_input.”
• That the raw PDFs (and associated text files) are the starting point.
• Preprocessing / Chunking Module:
• This is performed by notebooks such as extract_graph.ipynb, ner.ipynb, and related scripts.
• LLM-based Concept & Relationship Extraction:
• Interaction with the LLM (Mistral 7B OpenOrca) which is run locally via Ollama.
• Use of helper scripts (in the helpers folder including prompts.py) and the Ollama client (in the ollama directory).
• That this module takes text chunks as input and outputs extracted concepts and relationships with weighted edges.
• Graph Aggregation and Processing Module:
• That the output from extraction is further processed to aggregate relationships, calculate weights, deduplicate semantically similar concepts (with future improvements noted in the README), and also compute node degree and community detection.
• Mention use of Pandas for dataframes and NetworkX for graph creation.
• Data Output & Storage:
• “data_output” folder where CSV files like chunks.csv, concepts.csv, and graph.csv are stored.
• This module is responsible for saving the intermediate and final data artifacts.
• Visualization and Web Presentation:
• Visualization component that uses Pyvis to generate interactive JavaScript-based graph visualizations.
• That the final visual output is hosted (as seen in docs folder and linked via GitHub pages).
Relationships and flow between these components
• Start from “Data Input” feeding into the “Preprocessing” module.
• From “Preprocessing” pass the text chunks to the “LLM Extraction” module.
• The output from “LLM Extraction” is then directed into the “Graph Aggregation” module.
• Aggregated graph data is stored into CSV files in “Data Output.”
• Finally, the “Visualization” module reads from the aggregated graph data to render the interactive graph which is published as web pages (docs folder).
Important architectural patterns
• That the system is designed as a data processing pipeline with a clear sequential flow: ingestion → preprocessing → ML extraction → aggregation → visualization.
• Separate processing concerns clearly – for instance, the ML-powered extraction phase is isolated from the Graph Aggregation and the Visualization layers. This modular design simplifies debugging and future development (like adding deduplication or refining edge weight calculations as noted in the TODO comments of the README).
Relevant Technologies and Frameworks
• Python (overall language)
• Jupyter Notebooks for prototyping (extract_graph.ipynb and others)
• Ollama and Mistral 7B OpenOrca for local ML processing
• Pandas for data manipulation
• NetworkX for graph structuring
• Pyvis for visualization
• Poetry/pip (with environment.yml and pyproject.toml) for dependency management and local installation