# fareedkhan-dev/big-data-with-kg architecture

> Architecture diagram of the GitHub repository fareedkhan-dev/big-data-with-kg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fareedkhan-dev/big-data-with-kg
- Repository: https://github.com/fareedkhan-dev/big-data-with-kg

## Overview

##### Project Type and Purpose
- This is a data-processing pipeline implemented as a Jupyter Notebook.
- Its goal is to build a domain-specific Knowledge Graph (KG) from unstructured news articles about technology company acquisitions, leveraging both traditional NLP (spaCy, regex) and Large Language Models (via the Nebius/OpenAI API), then persisting, querying, and visualizing the KG.

##### High-Level Components
a. Data Sources
- CNN/DailyMail news dataset (via Hugging Face `datasets`)
- (Optionally) Other news APIs or file-based inputs
b. Preprocessing & ETL
- Data Acquisition module (`acquire_articles`)
- Data Cleaning & Normalization (`clean_article_text`)
c. Information Extraction
- Entity Exploration (spaCy NER analysis + plotting)
- LLM-driven Entity Type Selection
- LLM-based Named Entity Recognition
- LLM-based Relationship Extraction
d. KG Construction
- Entity Disambiguation & URI generation (`normalize_entity_text`)
- Schema/Ontology Alignment (mapping types to `schema.org` & custom `EX:`)
- Triple Generation (building an `rdflib.Graph`)
e. KG Refinement (Conceptual)
- Embedding Generation for entities (Nebius embedding API)
- Semantic similarity-based link exploration
- Placeholder for link-prediction model integration
f. Persistence & Utilization
- Graph serialization to Turtle (`.ttl`)
- SPARQL querying module
- Interactive visualization (PyVis)
- Generated artifacts in `graph_images/` (TTL and HTML viz)

##### Relationships & Data Flow
- Data flows from left to right:
1. **Data Source** → 2. **Ingestion & Cleaning** → 3. **IE (spaCy / LLM)** → 4. **KG Builder** → 5. **Refinement (Embeddings)** → 6. **Storage & Query** → 7. **Visualization / Output**

##### Architectural Patterns & Principles
- ETL/Pipeline pattern: clear separation of Extraction, Transformation, Loading phases.
- Modular design: each notebook cell/function implements a self-contained step.
- Hybrid AI: combine deterministic (spaCy, regex) and probabilistic (LLM) components.
- Layering: separate logical layers—Data Layer, Processing Layer, Semantic Layer, Storage Layer, Presentation Layer.

##### Technologies & Frameworks
- Python ecosystem: Jupyter Notebook, `datasets`, `spacy`, `rdflib`, `matplotlib`, `pyvis`
- External APIs: Nebius/OpenAI for LLM calls (text-gen and embeddings)
- Data artifacts: CNN/DailyMail dataset, Turtle file, interactive HTML visualization
- Query: SPARQL over `rdflib.Graph`

##### Essential Elements
- Arrowheads to indicate direction of data movement.
- Python modules & libraries
- External services & APIs
- File outputs (`.ttl`, `.html`)
- Version info (e.g., Notebook name: `code.ipynb`, README as reference).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers as sequential modules
    subgraph "Data Layer"
        DS["Data Sources\n- CNN/DailyMail\n- Other news APIs"]:::module
    end

    subgraph "Preprocessing & ETL"
        PRE["acquire_articles()\nclean_article_text()"]:::module
    end

    subgraph "Information Extraction"
        IE["spaCy NER exploration\nLLM-based NER/RE"]:::module
    end

    subgraph "KG Construction"
        KG["normalize_entity_text()\nschema mapping\nrdflib.Graph triple generation"]:::module
    end

    subgraph "Refinement & Enrichment"
        REF["Nebius/OpenAI embedding calls\nsemantic similarity exploration"]:::module
    end

    subgraph "Persistence & Querying"
        PERSIST["Turtle serialization\nSPARQL queries"]:::module
    end

    subgraph "Visualization"
        VIS["PyVis interactive graph rendering"]:::module
    end

    %% External Services
    HF["Hugging Face\ndatasets"]:::external
    NEB["Nebius/OpenAI API"]:::external

    %% Data Artifacts
    KGFile["tech_acquisitions_kg.ttl"]:::datastore
    VizHTML["tech_acquisitions_kg_interactive_viz.html"]:::datastore

    %% Documentation & Config
    README["README.md"]:::artifact
    REQ["requirements.txt"]:::artifact
    LICENSE["LICENSE"]:::artifact

    %% Data Flow
    HF -->|HTTP/HTTPS| DS
    DS --> PRE
    PRE --> IE
    IE -->|LLM calls| NEB
    IE --> KG
    KG --> REF
    REF -->|Embedding API| NEB
    REF --> PERSIST
    PERSIST --> KGFile
    PERSIST --> VIS
    VIS --> VizHTML

    %% Documentation connections
    README --> DS
    README --> VIS
    REQ --> DS
    LICENSE --> DS

    %% Click Events
    click DS "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/code.ipynb"
    click PRE "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/code.ipynb"
    click IE "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/code.ipynb"
    click KG "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/code.ipynb"
    click REF "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/code.ipynb"
    click PERSIST "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/code.ipynb"
    click VIS "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/code.ipynb"
    click KGFile "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/graph_images/tech_acquisitions_kg.ttl"
    click VizHTML "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/graph_images/tech_acquisitions_kg_interactive_viz.html"
    click README "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/README.md"
    click REQ "https://github.com/fareedkhan-dev/big-data-with-kg/blob/main/requirements.txt"
    click LICENSE "https://github.com/fareedkhan-dev/big-data-with-kg/tree/main/LICENSE"

    %% Styling
    classDef module fill:#a8d5e2,stroke:#333,stroke-width:1px
    classDef external fill:#f9d3b4,stroke-dasharray: 5 5
    classDef datastore fill:#c2f0c2,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2
    classDef artifact fill:#e2e2e2,stroke:#666,stroke-width:1px
```
