# tractorjuice/knowledge-graph-llms architecture

> Architecture diagram of the GitHub repository tractorjuice/knowledge-graph-llms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tractorjuice/knowledge-graph-llms
- Repository: https://github.com/tractorjuice/knowledge-graph-llms

## Overview

##### Project Type and Purpose
- This is a full-stack knowledge-graph generation and visualization application.
- The “backend” is a Streamlit-based Python pipeline that handles text ingestion, chunking, LLM calls, graph construction, and multi-format export.
- The “frontend” consists of two parts:
- Streamlit UI (Python) for input and on-the-fly graph generation
- Standalone JavaScript app (js-app) for interactive graph exploration and export

##### Top-Level Components
- Streamlit Web Application
- Text Processing & LLM Pipeline
- Graph Generation & Export Module
- OpenAI GPT-4o External Service
- File Storage (temporary export files: HTML, JSON, GraphML, GML)
- JavaScript Graph Explorer (client-side static web app)

##### Component Breakdown
A. Streamlit Frontend (streamlit_app.py, .streamlit/config.toml)
- Sidebar controls for upload/text input
- Of progress bar, status messages
- Embeds PyVis output or download links

B. Text Ingestion & Chunking (generate_knowledge_graph.py + built-in in streamlit_app)
- Detects .md vs .txt
- Primary semantic split by markdown headers
- Fallback character-based splitting with overlap
- Ensures ≤100k token chunks

C. Entity-Relationship Extraction (LangChain + OpenAI)
- LLMGraphTransformer calls GPT-4o via LangChain
- Parallel processing of chunks
- Progress tracking hooks for Streamlit UI

D. Graph Construction & Visualization (PyVis + networkx)
- Constructs NetworkX graph object
- Applies ForceAtlas2 physics, dark theme
- Generates HTML visualization or returns graph data structure

E. Multi-format Exporters
- HTML (PyVis)
- JSON (standard)
- GraphML (XML-based)
- GML (Graph Modeling Language)
- Exports written to disk, referenced by Streamlit for download

F. JavaScript Graph Explorer (js-app/)
- index.html + app.js, dataLoader.js, graphController.js, uiController.js, exportController.js
- Loads JSON/GraphML/GML via drag & drop or quick-load buttons
- Provides advanced filtering, search, PNG/CSV export
- Entirely client-side—no server required

G. External Dependencies & Configuration
- OpenAI API (key from .env)
- Python packages: langchain, langchain-experimental, pyvis, streamlit, networkx, tiktoken, python-dotenv
- .devcontainer for consistent dev environment

##### Data Flow and Interactions
1. User opens Streamlit UI → chooses file or pastes text
2. Streamlit passes raw text to Chunking component
3. Each chunk → LangChain LLMGraphTransformer → calls OpenAI GPT-4o → returns entities/relations
4. Collected triples → NetworkX graph → PyVis HTML or raw graph data
5. Streamlit displays PyVis graph inline and exposes download links for HTML/JSON/GraphML/GML
6. JavaScript Graph Explorer loads exported file → renders interactive view → allows filtering/export

##### Architectural Patterns and Principles
- Separation of Concerns: UI, processing pipeline, graph engine, export, and explorer are distinct modules
- Pipeline Modularity: Chunking, extraction, and export stages can be extended or replaced independently
- File-based Integration: Streamlit and JS app share data via standard graph file formats
- External Service Encapsulation: OpenAI calls are managed through LangChain for unified LLM interfacing

##### Critical Details to Capture
- Semantic chunking logic and token limits
- LangChain’s role as LLM orchestrator
- PyVis force-directed graph as the default visualization engine
- Static file exports enabling offline, client-side exploration
- Progress tracking integration between pipeline and UI

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    User["User"]:::actor
    User -->|"opens Streamlit UI"| StreamlitUI

    %% Streamlit Frontend
    subgraph "Streamlit Frontend"
        StreamlitUI["streamlit_app.py"]:::ui
        Config[".streamlit/config.toml"]:::config
    end
    click StreamlitUI "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/streamlit_app.py"
    click Config "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/.streamlit/config.toml"

    User -->|"uploads file or pastes text"| StreamlitUI

    %% Processing Pipeline
    subgraph "Python Processing Pipeline" 
        Chunker["Text Ingestion & Chunking\n(generate_knowledge_graph.py)"]:::process
        Extractor["Entity-Relationship Extraction\n(generate_knowledge_graph.py)"]:::process
        GraphGen["Graph Construction & Visualization\n(generate_knowledge_graph.py)"]:::process
        Exporter["Multi-format Exporters\n(generate_knowledge_graph.py)"]:::export
    end
    click Chunker "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/generate_knowledge_graph.py"
    click Extractor "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/generate_knowledge_graph.py"
    click GraphGen "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/generate_knowledge_graph.py"
    click Exporter "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/generate_knowledge_graph.py"

    StreamlitUI -->|"raw text"| Chunker
    Chunker -->|"chunks"| Extractor
    Extractor -->|"triples"| GraphGen
    GraphGen -->|"graph data/HTML"| Exporter
    Exporter -->|"files"| FileStore

    %% External LLM Service
    OpenAI["OpenAI GPT-4o"]:::external
    Extractor -->|"calls LLM"| OpenAI

    %% File Storage
    FileStore["File Storage\n(HTML, JSON, GraphML, GML)"]:::datastore

    %% JavaScript Graph Explorer
    subgraph "JavaScript Graph Explorer (js-app)"
        JSIndex["index.html"]:::ui
        AppJS["js/app.js"]:::ui
        DataLoader["js/dataLoader.js"]:::ui
        GraphCtrl["js/graphController.js"]:::ui
        UICtrl["js/uiController.js"]:::ui
        ExportCtrl["js/exportController.js"]:::ui
        GraphData["js/graphData.js"]:::ui
        Styles["styles.css"]:::ui
        README["README.md"]:::doc
    end
    click JSIndex "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/index.html"
    click AppJS "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/js/app.js"
    click DataLoader "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/js/dataLoader.js"
    click GraphCtrl "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/js/graphController.js"
    click UICtrl "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/js/uiController.js"
    click ExportCtrl "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/js/exportController.js"
    click GraphData "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/js/graphData.js"
    click Styles "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/styles.css"
    click README "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/js-app/README.md"

    FileStore -->|"loads files"| JSIndex
    JSIndex --> AppJS
    AppJS --> DataLoader
    AppJS --> GraphCtrl
    AppJS --> UICtrl
    AppJS --> ExportCtrl

    %% Configuration & Environment
    subgraph "Configuration & Environment"
        Env1[".claude/settings.local.json"]:::config
        Env2[".devcontainer/devcontainer.json"]:::config
        Req["requirements.txt"]:::config
        Doc1["README.md"]:::doc
        Doc2["CHANGELOG.md"]:::doc
        Doc3["LICENSE"]:::doc
        Doc4["VERSION"]:::doc
        Doc5["CLAUDE.md"]:::doc
    end
    click Env1 "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/.claude/settings.local.json"
    click Env2 "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/.devcontainer/devcontainer.json"
    click Req "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/requirements.txt"
    click Doc1 "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/README.md"
    click Doc2 "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/CHANGELOG.md"
    click Doc3 "https://github.com/tractorjuice/knowledge-graph-llms/tree/main/LICENSE"
    click Doc4 "https://github.com/tractorjuice/knowledge-graph-llms/tree/main/VERSION"
    click Doc5 "https://github.com/tractorjuice/knowledge-graph-llms/blob/main/CLAUDE.md"

    %% Data Flow to UI
    Exporter -->|"download links"| StreamlitUI

    %% Styles
    classDef actor fill:#FFD580,stroke:#E69F00;
    classDef ui fill:#A7D3F2,stroke:#1E7FCB;
    classDef process fill:#FFE1E1,stroke:#E03E3E;
    classDef export fill:#E6E6FA,stroke:#7B68EE;
    classDef external fill:#D5F5E3,stroke:#27AE60,stroke-dasharray: 5 5;
    classDef datastore fill:#FFFACD,stroke:#F1C40F;
    classDef config fill:#F0F0F0,stroke:#BFBFBF,stroke-dasharray: 2 2;
    classDef doc fill:#F5E1DA,stroke:#D35400;
```
