How tractorjuice/knowledge-graph-llms works
Project Type and Purpose
• This is a full-stack knowledge-graph generation and visualization application.
• The “backend” is a Streamlit-based Python pipeline that handles text ingestion, chunking, LLM calls, graph construction, and multi-format export.
• The “frontend” consists of two parts:
• Streamlit UI (Python) for input and on-the-fly graph generation
• Standalone JavaScript app (js-app) for interactive graph exploration and export
Top-Level Components
• Streamlit Web Application
• Text Processing & LLM Pipeline
• Graph Generation & Export Module
• OpenAI GPT-4o External Service
• File Storage (temporary export files: HTML, JSON, GraphML, GML)
• JavaScript Graph Explorer (client-side static web app)
Component Breakdown
A. Streamlit Frontend (streamlit_app.py, .streamlit/config.toml)
• Sidebar controls for upload/text input
• Of progress bar, status messages
• Embeds PyVis output or download links
B. Text Ingestion & Chunking (generate_knowledge_graph.py + built-in in streamlit_app)
• Detects .md vs .txt
• Primary semantic split by markdown headers
• Fallback character-based splitting with overlap
• Ensures ≤100k token chunks
C. Entity-Relationship Extraction (LangChain + OpenAI)
• LLMGraphTransformer calls GPT-4o via LangChain
• Parallel processing of chunks
• Progress tracking hooks for Streamlit UI
D. Graph Construction & Visualization (PyVis + networkx)
• Constructs NetworkX graph object
• Applies ForceAtlas2 physics, dark theme
• Generates HTML visualization or returns graph data structure
E. Multi-format Exporters
• HTML (PyVis)
• JSON (standard)
• GraphML (XML-based)
• GML (Graph Modeling Language)
• Exports written to disk, referenced by Streamlit for download
F. JavaScript Graph Explorer (js-app/)
• index.html + app.js, dataLoader.js, graphController.js, uiController.js, exportController.js
• Loads JSON/GraphML/GML via drag & drop or quick-load buttons
• Provides advanced filtering, search, PNG/CSV export
• Entirely client-side—no server required
G. External Dependencies & Configuration
• OpenAI API (key from .env)
• Python packages: langchain, langchain-experimental, pyvis, streamlit, networkx, tiktoken, python-dotenv
• .devcontainer for consistent dev environment
Data Flow and Interactions
1. User opens Streamlit UI → chooses file or pastes text
2. Streamlit passes raw text to Chunking component
3. Each chunk → LangChain LLMGraphTransformer → calls OpenAI GPT-4o → returns entities/relations
4. Collected triples → NetworkX graph → PyVis HTML or raw graph data
5. Streamlit displays PyVis graph inline and exposes download links for HTML/JSON/GraphML/GML
6. JavaScript Graph Explorer loads exported file → renders interactive view → allows filtering/export
Architectural Patterns and Principles
• Separation of Concerns: UI, processing pipeline, graph engine, export, and explorer are distinct modules
• Pipeline Modularity: Chunking, extraction, and export stages can be extended or replaced independently
• File-based Integration: Streamlit and JS app share data via standard graph file formats
• External Service Encapsulation: OpenAI calls are managed through LangChain for unified LLM interfacing
Critical Details to Capture
• Semantic chunking logic and token limits
• LangChain’s role as LLM orchestrator
• PyVis force-directed graph as the default visualization engine
• Static file exports enabling offline, client-side exploration
• Progress tracking integration between pipeline and UI