# rahulnyk/knowledge_graph architecture

> Architecture diagram of the GitHub repository rahulnyk/knowledge_graph, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rahulnyk/knowledge_graph
- Repository: https://github.com/rahulnyk/knowledge_graph

## Overview

- Data Input & Ingestion Module:
- Directories in “data_input.”
- That the raw PDFs (and associated text files) are the starting point.
- Preprocessing / Chunking Module:
- This is performed by notebooks such as extract_graph.ipynb, ner.ipynb, and related scripts.
- LLM-based Concept & Relationship Extraction:
- Interaction with the LLM (Mistral 7B OpenOrca) which is run locally via Ollama.
- Use of helper scripts (in the helpers folder including prompts.py) and the Ollama client (in the ollama directory).
- That this module takes text chunks as input and outputs extracted concepts and relationships with weighted edges.
- Graph Aggregation and Processing Module:
- That the output from extraction is further processed to aggregate relationships, calculate weights, deduplicate semantically similar concepts (with future improvements noted in the README), and also compute node degree and community detection.
- Mention use of Pandas for dataframes and NetworkX for graph creation.
- Data Output & Storage:
- “data_output” folder where CSV files like chunks.csv, concepts.csv, and graph.csv are stored.
- This module is responsible for saving the intermediate and final data artifacts.
- Visualization and Web Presentation:
- Visualization component that uses Pyvis to generate interactive JavaScript-based graph visualizations.
- That the final visual output is hosted (as seen in docs folder and linked via GitHub pages).

##### Relationships and flow between these components
- Start from “Data Input” feeding into the “Preprocessing” module.
- From “Preprocessing” pass the text chunks to the “LLM Extraction” module.
- The output from “LLM Extraction” is then directed into the “Graph Aggregation” module.
- Aggregated graph data is stored into CSV files in “Data Output.”
- Finally, the “Visualization” module reads from the aggregated graph data to render the interactive graph which is published as web pages (docs folder).

##### Important architectural patterns
- That the system is designed as a data processing pipeline with a clear sequential flow: ingestion → preprocessing → ML extraction → aggregation → visualization.
- Separate processing concerns clearly – for instance, the ML-powered extraction phase is isolated from the Graph Aggregation and the Visualization layers. This modular design simplifies debugging and future development (like adding deduplication or refining edge weight calculations as noted in the TODO comments of the README).

##### Relevant Technologies and Frameworks
- Python (overall language)
- Jupyter Notebooks for prototyping (extract_graph.ipynb and others)
- Ollama and Mistral 7B OpenOrca for local ML processing
- Pandas for data manipulation
- NetworkX for graph structuring
- Pyvis for visualization
- Poetry/pip (with environment.yml and pyproject.toml) for dependency management and local installation

## Diagram (Mermaid)

```mermaid
flowchart TD
    I["Dependency Management (env config)"]
    A["Data Input & Ingestion"]
    B1["Extract Graph Notebook (Preprocessing)"]
    B2["NER Notebook (Preprocessing)"]
    C1["Ollama Client (LLM Extraction)"]
    C2["Prompts Helper (LLM Extraction)"]
    C3["Helpers Init (LLM Extraction)"]
    C4["Ollama Init (LLM Extraction)"]
    
    subgraph "Graph Aggregation and Processing"
        D["Graph Aggregation (Pandas,NetworkX)"]
        E["Aggregation Notebook"]
        F["Dataframe Helper"]
    end
    G["Data Output & Storage (CSV)"]
    H["Interactive Visualization (Pyvis)"]
    
    %% Data Flow
    I --> A
    A --> B1
    A --> B2
    B1 --> C1
    B2 --> C1
    C2 --> C1
    C3 --> C1
    C4 --> C1
    C1 --> D
    E --> D
    F --> D
    D --> G
    G --> H

    %% Click Events
    click A "https://github.com/rahulnyk/knowledge_graph/tree/main/data_input/"
    click B1 "https://github.com/rahulnyk/knowledge_graph/blob/main/extract_graph.ipynb"
    click B2 "https://github.com/rahulnyk/knowledge_graph/blob/main/ner.ipynb"
    click C1 "https://github.com/rahulnyk/knowledge_graph/blob/main/ollama/client.py"
    click C2 "https://github.com/rahulnyk/knowledge_graph/blob/main/helpers/prompts.py"
    click C3 "https://github.com/rahulnyk/knowledge_graph/blob/main/helpers/__init__.py"
    click C4 "https://github.com/rahulnyk/knowledge_graph/blob/main/ollama/__init__.py"
    click E "https://github.com/rahulnyk/knowledge_graph/blob/main/old_notebooks/concept_graph.ipynb"
    click F "https://github.com/rahulnyk/knowledge_graph/blob/main/helpers/df_helpers.py"
    click G "https://github.com/rahulnyk/knowledge_graph/tree/main/data_output/"
    click H "https://github.com/rahulnyk/knowledge_graph/blob/main/docs/graph.html"

    %% Style Definitions
    classDef dependency fill:#e0e0e0,stroke:#333,stroke-width:2px;
    classDef ingestion fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef preprocessing fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef extraction fill:#ffeb3b,stroke:#000,stroke-width:2px;
    classDef aggregation fill:#ffa500,stroke:#000,stroke-width:2px;
    classDef storage fill:#dda0dd,stroke:#000,stroke-width:2px;
    classDef visualization fill:#ff69b4,stroke:#000,stroke-width:2px;
    
    class I dependency;
    class A ingestion;
    class B1,B2 preprocessing;
    class C1,C2,C3,C4 extraction;
    class D,E,F aggregation;
    class G storage;
    class H visualization;
```
