How uds-lsv/stereokg works
To accurately capture the architecture of StereoKG, break the system into logical layers and highlight key components, data flows, and external dependencies. Follow these steps tailored to this project’s purpose—a pipeline that scrapes social media, processes text into a knowledge graph, and uses that KG for language‐model experiments.
Main Components
• Configuration Layer
– Credentials.py (API keys, directory paths, subject entities list)
– questions/… (one file per entity containing question templates)
• Ingestion Layer (Data Extraction)
– scrape_reddit.py (uses PRAW to pull Reddit posts)
– scrape_twitter.py (uses Twitter API via credentials)
• Transformation Layer (Data Processing)
– StatementMaker, question_to_statement, inflection (convert raw text to canonical sentences)
– fast_clustering (group similar sentences)
– triple_generation (OpenIE wrapper to extract candidate triples)
– triple_selection (DistilBERT-CoLA model to pick best triples)
• Storage Layer (Knowledge Graph)
– kg/stereoKG.tsv (structured triples)
– kg/stereoKG_linearised.txt (verbalized triples via triple_to_text model)
• Experimental Layer (LM Integration)
– dataset_creation (create_interlm_data.py, weblg_generate_dataset.py)
– model_training (hs_detection.py, intermediate_pretraining.py, triple_to_text.py)
– Outputs: fine-tuned models (StereoKG-DT-SK, StereoKG-DT-UK on HuggingFace)
• External Services & Libraries
– Reddit API, Twitter Developer API
– OpenIE (for triple extraction)
– DistilBERT-CoLA, SimpleTransformers, HuggingFace Transformers, PyTorch, NLTK
Relationships and Data Flow
• Credentials.py feeds API keys and file paths into scrape_reddit.py and scrape_twitter.py.
• Raw text flows into data_processing scripts in sequence: conversion → clustering → triple extraction → triple selection.
• Selected triples are written to kg/stereoKG.tsv.
• triple_to_text (in both data_processing and lm_integration) converts triples to natural language, producing stereoKG_linearised.txt.
• The KG (structured & linearized) is consumed by lm_integration/dataset_creation to build training corpora.
• Datasets feed into model_training scripts, culminating in new model artifacts.
Architectural Patterns & Principles
• Pipeline/Batch Processing: clear stages from ingestion through transformation to output.
• Modular Scripts: each script handles one responsibility (single‐responsibility principle).
• Configuration Isolation: Credentials.py and questions folder strictly hold metadata and templates.
• Reusability: triple_to_text is used both for KG linearization and for LM experiment data preparation.