How fareedkhan-dev/kg-pipeline works
Project type and purpose
• This is a single‐notebook data‐processing pipeline (code.ipynb) that converts raw unstructured text into an interactive knowledge graph, using an LLM (via OpenAI‐compatible API), NetworkX, and ipycytoscape. It is neither a client–server web app nor a library, but an end‐to‐end NLP/data‐visualization workflow.
File structure
• Top level contains only README.md (documentation) and code.ipynb (the actual pipeline).
• All logic lives in one Jupyter notebook: setup, chunking, LLM calls, JSON parsing, normalization, NetworkX graph construction, and ipycytoscape visualization.
• External dependency files are managed at runtime via pip commands in the notebook.
Extract architectural insights from the README
• The workflow is strictly sequential:
1) Install & import libraries
2) Configure LLM credentials
3) & chunk raw text
4) Send chunks to LLM for SPO extraction
5) Parse & normalize JSON triples
6) Build the graph with NetworkX
• Key external service: the OpenAI-compatible LLM API (could be OpenAI, Ollama, Nebius, etc.).
Main components and their interactions
a. Components
• Jupyter Notebook Environment (Python runtime)
• Chunking Module (splits text into overlapping word‐based blocks)
• LLM Client Module (openai.OpenAI client)
• Prompt Templates (system + user prompts for SPO extraction)
• JSON Parser & Fallback Logic (regex, JSON decoding)
• Normalization & Deduplication Module
• Graph Builder (NetworkX DiGraph)
• Visualization Layer (ipycytoscape CytoscapeWidget)
• External LLM API Service
b. Data Flow
Raw Text → Chunking → LLM API calls (via Prompt Templates) → JSON SPO triples → Parsing Module → Normalization & De‐duplication → NetworkX Graph → ipycytoscape Data → Interactive Visualization
c. Architectural Patterns
• Pipeline/ETL pattern: each stage transforms data
• Separation of concerns: text preparation, extraction, parsing, graph construction, visualization
d. Key Technologies
• Python & Jupyter Notebook
• openai (LLM client)
• networkx (graph construction)
• ipycytoscape & ipywidgets (interactive visualization)
• pandas (data inspection)
Detailed elements
• If desired, notes on configurable parameters (chunk_size, overlap, llm_temperature, max_tokens) near the relevant module.