# thegreatvoid/learnkg architecture

> Architecture diagram of the GitHub repository thegreatvoid/learnkg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thegreatvoid/learnkg
- Repository: https://github.com/thegreatvoid/learnkg

## Overview

This repository turns a text corpus into an interactive, directed knowledge graph. The principal workflow loads and chunks documents, asks a local LLM for directional concept relations, adds same-chunk proximity links, combines the relations into a weighted graph, and renders it for browser exploration. README also describes community coloring and graph analysis. The sampled pipeline supports the main processing and visualization links; some intermediate pipeline code was omitted, so those stage boundaries are based on the README’s documented workflow.

Main entry points: [`run_pipeline.py`](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py) (Pipeline runner), [`helpers/df_helpers.py`](https://github.com/thegreatvoid/learnkg/blob/main/helpers/df_helpers.py) (Chunk records), [`run_directed.py`](https://github.com/thegreatvoid/learnkg/blob/main/run_directed.py) (Directed graph renderer), [`helpers/prompts.py`](https://github.com/thegreatvoid/learnkg/blob/main/helpers/prompts.py) (Relation prompting)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Corpus processing"]
  node_pipeline["Pipeline runner<br/>[run_pipeline.py]"]
  node_documents["Document loading<br/>[run_pipeline.py]"]
  node_chunks["Chunk records<br/>[df_helpers.py]"]
end

subgraph group_extract["Relation extraction"]
  node_prompt["Relation prompting<br/>[prompts.py]"]
  node_ollama_client["Ollama client<br/>[client.py]"]
  node_relations["Relation dataframe<br/>[df_helpers.py]"]
end

subgraph group_graph["Graph construction"]
  node_proximity["Proximity edges<br/>[run_pipeline.py]"]
  node_merge["Edge consolidation<br/>[run_pipeline.py]"]
  node_directed_graph["Directed graph<br/>[run_pipeline.py]"]
  node_graph_csv["Precomputed graph<br/>[run_directed.py]"]
end

subgraph group_display["Analysis and display"]
  node_communities["Community analysis<br/>[run_pipeline.py]"]
  node_ranking["PageRank sizing<br/>[run_pipeline.py]"]
  node_renderer["Interactive renderer<br/>[run_pipeline.py]"]
  node_standalone_renderer["Directed graph renderer<br/>[run_directed.py]"]
  node_html["Browser graph<br/>[run_directed.py]"]
end

node_user(("Corpus user"))
node_ollama_service["Ollama model service"]

node_user -->|"starts"| node_pipeline
node_pipeline -->|"loads corpus"| node_documents
node_documents -->|"splits text"| node_chunks
node_pipeline -->|"converts pages"| node_chunks
node_chunks -->|"extracts relations"| node_prompt
node_prompt -->|"requests generation"| node_ollama_client
node_ollama_client -->|"posts prompt"| node_ollama_service
node_chunks -->|"builds relation rows"| node_relations
node_pipeline -->|"converts extracted graph"| node_relations
node_pipeline -->|"calculates co-occurrence"| node_proximity
node_pipeline -->|"groups and sums edges"| node_merge
node_pipeline -->|"builds DiGraph"| node_directed_graph
node_pipeline -->|"detects communities"| node_communities
node_pipeline -->|"sizes nodes"| node_ranking
node_pipeline -->|"renders graph"| node_renderer
node_user -->|"runs renderer"| node_standalone_renderer
node_graph_csv -->|"loads graph data"| node_standalone_renderer
node_standalone_renderer -->|"writes visualization"| node_html

click node_pipeline "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_documents "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_chunks "https://github.com/thegreatvoid/learnkg/blob/main/helpers/df_helpers.py"
click node_prompt "https://github.com/thegreatvoid/learnkg/blob/main/helpers/prompts.py"
click node_ollama_client "https://github.com/thegreatvoid/learnkg/blob/main/ollama/client.py"
click node_relations "https://github.com/thegreatvoid/learnkg/blob/main/helpers/df_helpers.py"
click node_proximity "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_merge "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_directed_graph "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_communities "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_ranking "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_renderer "https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py"
click node_graph_csv "https://github.com/thegreatvoid/learnkg/blob/main/run_directed.py"
click node_standalone_renderer "https://github.com/thegreatvoid/learnkg/blob/main/run_directed.py"
click node_html "https://github.com/thegreatvoid/learnkg/blob/main/run_directed.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline,node_documents,node_chunks,node_user toneBlue
class node_prompt,node_ollama_client,node_relations toneAmber
class node_proximity,node_merge,node_directed_graph,node_graph_csv toneMint
class node_communities,node_ranking,node_renderer,node_standalone_renderer,node_html toneRose
class node_ollama_service toneIndigo
```

## Components

### Corpus processing

- [Pipeline runner](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [Document loading](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [Chunk records](https://github.com/thegreatvoid/learnkg/blob/main/helpers/df_helpers.py): `helpers/df_helpers.py`

### Relation extraction

- [Relation prompting](https://github.com/thegreatvoid/learnkg/blob/main/helpers/prompts.py): `helpers/prompts.py`
- [Ollama client](https://github.com/thegreatvoid/learnkg/blob/main/ollama/client.py): `ollama/client.py`
- [Relation dataframe](https://github.com/thegreatvoid/learnkg/blob/main/helpers/df_helpers.py): `helpers/df_helpers.py`

### Graph construction

- [Proximity edges](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [Edge consolidation](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [Directed graph](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [Precomputed graph](https://github.com/thegreatvoid/learnkg/blob/main/run_directed.py): `run_directed.py`

### Analysis and display

- [Community analysis](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [PageRank sizing](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [Interactive renderer](https://github.com/thegreatvoid/learnkg/blob/main/run_pipeline.py): `run_pipeline.py`
- [Directed graph renderer](https://github.com/thegreatvoid/learnkg/blob/main/run_directed.py): `run_directed.py`
- [Browser graph](https://github.com/thegreatvoid/learnkg/blob/main/run_directed.py): `run_directed.py`

### Other components

- **Corpus user**
- **Ollama model service**

## Connections

- Corpus user → Pipeline runner: starts (evidence: `README.md`)
- Pipeline runner → Document loading: loads corpus (evidence: `README.md`)
- Document loading → Chunk records: splits text (evidence: `README.md`)
- Pipeline runner → Chunk records: converts pages (evidence: `run_pipeline.py`)
- Chunk records → Relation prompting: extracts relations (evidence: `helpers/df_helpers.py`)
- Relation prompting → Ollama client: requests generation (evidence: `helpers/prompts.py`)
- Ollama client → Ollama model service: posts prompt (evidence: `ollama/client.py`)
- Chunk records → Relation dataframe: builds relation rows (evidence: `helpers/df_helpers.py`)
- Pipeline runner → Relation dataframe: converts extracted graph (evidence: `run_pipeline.py`)
- Pipeline runner → Proximity edges: calculates co-occurrence (evidence: `run_pipeline.py`)
- Pipeline runner → Edge consolidation: groups and sums edges (evidence: `run_pipeline.py`)
- Pipeline runner → Directed graph: builds DiGraph (evidence: `run_pipeline.py`)
- Pipeline runner → Community analysis: detects communities (evidence: `run_pipeline.py`)
- Pipeline runner → PageRank sizing: sizes nodes (evidence: `run_pipeline.py`)
- Pipeline runner → Interactive renderer: renders graph (evidence: `run_pipeline.py`)
- Corpus user → Directed graph renderer: runs renderer (evidence: `run_directed.py`)
- Precomputed graph → Directed graph renderer: loads graph data (evidence: `run_directed.py`)
- Directed graph renderer → Browser graph: writes visualization (evidence: `run_directed.py`)
