How fareedkhan-dev/big-data-with-kg works
Project Type and Purpose
• This is a data-processing pipeline implemented as a Jupyter Notebook.
• Its goal is to build a domain-specific Knowledge Graph (KG) from unstructured news articles about technology company acquisitions, leveraging both traditional NLP (spaCy, regex) and Large Language Models (via the Nebius/OpenAI API), then persisting, querying, and visualizing the KG.
High-Level Components
a. Data Sources
• CNN/DailyMail news dataset (via Hugging Face datasets)
• (Optionally) Other news APIs or file-based inputs
b. Preprocessing & ETL
• Data Acquisition module (acquire_articles)
• Data Cleaning & Normalization (clean_article_text)
c. Information Extraction
• Entity Exploration (spaCy NER analysis + plotting)
• LLM-driven Entity Type Selection
• LLM-based Named Entity Recognition
• LLM-based Relationship Extraction
d. KG Construction
• Entity Disambiguation & URI generation (normalize_entity_text)
• Schema/Ontology Alignment (mapping types to schema.org & custom EX:)
• Triple Generation (building an rdflib.Graph)
e. KG Refinement (Conceptual)
• Embedding Generation for entities (Nebius embedding API)
• Semantic similarity-based link exploration
• Placeholder for link-prediction model integration
f. Persistence & Utilization
• Graph serialization to Turtle (.ttl)
• SPARQL querying module
• Interactive visualization (PyVis)
• Generated artifacts in graph_images/ (TTL and HTML viz)
Relationships & Data Flow
• Data flows from left to right:
1. Data Source → 2. Ingestion & Cleaning → 3. IE (spaCy / LLM) → 4. KG Builder → 5. Refinement (Embeddings) → 6. Storage & Query → 7. Visualization / Output
Architectural Patterns & Principles
• ETL/Pipeline pattern: clear separation of Extraction, Transformation, Loading phases.
• Modular design: each notebook cell/function implements a self-contained step.
• Hybrid AI: combine deterministic (spaCy, regex) and probabilistic (LLM) components.
• Layering: separate logical layers—Data Layer, Processing Layer, Semantic Layer, Storage Layer, Presentation Layer.
Technologies & Frameworks
• Python ecosystem: Jupyter Notebook, datasets, spacy, rdflib, matplotlib, pyvis
• External APIs: Nebius/OpenAI for LLM calls (text-gen and embeddings)
• Data artifacts: CNN/DailyMail dataset, Turtle file, interactive HTML visualization
• Query: SPARQL over rdflib.Graph
Essential Elements
• Arrowheads to indicate direction of data movement.
• Python modules & libraries
• External services & APIs
• File outputs (.ttl, .html)
• Version info (e.g., Notebook name: code.ipynb, README as reference).