# fareedkhan-dev/kg-pipeline architecture

> Architecture diagram of the GitHub repository fareedkhan-dev/kg-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fareedkhan-dev/kg-pipeline
- Repository: https://github.com/fareedkhan-dev/kg-pipeline

## Overview

##### Project type and purpose
- This is a single‐notebook data‐processing pipeline (code.ipynb) that converts raw unstructured text into an interactive knowledge graph, using an LLM (via OpenAI‐compatible API), NetworkX, and ipycytoscape. It is neither a client–server web app nor a library, but an end‐to‐end NLP/data‐visualization workflow.

##### File structure
- Top level contains only README.md (documentation) and code.ipynb (the actual pipeline).
- All logic lives in one Jupyter notebook: setup, chunking, LLM calls, JSON parsing, normalization, NetworkX graph construction, and ipycytoscape visualization.
- External dependency files are managed at runtime via pip commands in the notebook.

##### Extract architectural insights from the README
- The workflow is strictly sequential:
1) Install & import libraries
2) Configure LLM credentials
3) & chunk raw text
4) Send chunks to LLM for SPO extraction
5) Parse & normalize JSON triples
6) Build the graph with NetworkX
- Key external service: the OpenAI-compatible LLM API (could be OpenAI, Ollama, Nebius, etc.).

##### Main components and their interactions
a. Components
- Jupyter Notebook Environment (Python runtime)
- Chunking Module (splits text into overlapping word‐based blocks)
- LLM Client Module (openai.OpenAI client)
- Prompt Templates (system + user prompts for SPO extraction)
- JSON Parser & Fallback Logic (regex, JSON decoding)
- Normalization & Deduplication Module
- Graph Builder (NetworkX DiGraph)
- Visualization Layer (ipycytoscape CytoscapeWidget)
- External LLM API Service
b. Data Flow
Raw Text → Chunking → LLM API calls (via Prompt Templates) → JSON SPO triples → Parsing Module → Normalization & De‐duplication → NetworkX Graph → ipycytoscape Data → Interactive Visualization
c. Architectural Patterns
- Pipeline/ETL pattern: each stage transforms data
- Separation of concerns: text preparation, extraction, parsing, graph construction, visualization
d. Key Technologies
- Python & Jupyter Notebook
- openai (LLM client)
- networkx (graph construction)
- ipycytoscape & ipywidgets (interactive visualization)
- pandas (data inspection)

##### Detailed elements
- If desired, notes on configurable parameters (chunk_size, overlap, llm_temperature, max_tokens) near the relevant module.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Notebook Pipeline"
        JN["Jupyter Notebook Environment<br>(Python & Jupyter)"]:::module
        CT["Chunking Module<br>(pure Python)"]:::module
        LC["LLM Client Module<br>(openai.OpenAI)"]:::module
        PT["Prompt Templates<br>(system + user)"]:::module
        PS["JSON Parser & Fallback Logic<br>(json + regex)"]:::module
        ND["Normalization & Deduplication Module"]:::module
        GB["Graph Builder<br>(NetworkX DiGraph)"]:::module
        VZ["Visualization Layer<br>(ipycytoscape CytoscapeWidget)"]:::viz
    end

    RT[(Raw Text)]:::data
    Chunks[(Text Chunks)]:::data
    JSON[(JSON SPO Triples)]:::data
    Triples[(Clean Triples)]:::data
    NXG[(NetworkX Graph)]:::data

    EXT["External LLM API Service"]:::external
    ENV["Environment Variables<br>(API Keys & Config)"]:::module
    DOC["External Documentation<br>(Project Purpose & Setup)"]:::data

    %% Data Flow
    RT -->|raw_text| CT
    CT -->|text_chunks| Chunks
    Chunks -->|chunk_payload| LC
    PT -->|prompts| LC
    ENV -->|credentials| LC
    LC -->|API calls| EXT
    EXT -->|raw_JSON| LC
    LC -->|raw_JSON| JSON
    JSON -->|parsed_JSON| PS
    PS -->|clean_triples| Triples
    Triples -->|triples_payload| GB
    GB -->|NetworkX_graph| NXG
    NXG -->|graph_data| VZ

    %% Notebook orchestration
    JN --> CT
    JN --> PT
    JN --> ENV
    JN --> PS
    JN --> ND
    JN --> GB
    JN --> VZ

    %% Documentation link
    DOC

    %% Click Events
    click JN "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click CT "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click LC "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click PT "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click PS "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click ND "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click GB "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click VZ "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/code.ipynb"
    click DOC "https://github.com/fareedkhan-dev/kg-pipeline/blob/main/README.md"

    %% Styles
    classDef module fill:#D0E6FF,stroke:#0366d6,color:#0366d6;
    classDef data fill:#E6FFEA,stroke:#28A745,color:#1e4620;
    classDef external fill:#FFE8D6,stroke:#D64826,color:#872E1A;
    classDef viz fill:#F0E6FF,stroke:#8C2DCF,color:#58157C;
```
