# uds-lsv/stereokg architecture

> Architecture diagram of the GitHub repository uds-lsv/stereokg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uds-lsv/stereokg
- Repository: https://github.com/uds-lsv/stereokg

## Overview

To accurately capture the architecture of StereoKG, break the system into logical layers and highlight key components, data flows, and external dependencies. Follow these steps tailored to this project’s purpose—a pipeline that scrapes social media, processes text into a knowledge graph, and uses that KG for language‐model experiments.

##### Main Components
- Configuration Layer
– Credentials.py (API keys, directory paths, subject entities list)
– questions/… (one file per entity containing question templates)
- Ingestion Layer (Data Extraction)
– scrape_reddit.py (uses PRAW to pull Reddit posts)
– scrape_twitter.py (uses Twitter API via credentials)
- Transformation Layer (Data Processing)
– StatementMaker, question_to_statement, inflection (convert raw text to canonical sentences)
– fast_clustering (group similar sentences)
– triple_generation (OpenIE wrapper to extract candidate triples)
– triple_selection (DistilBERT-CoLA model to pick best triples)
- Storage Layer (Knowledge Graph)
– kg/stereoKG.tsv (structured triples)
– kg/stereoKG_linearised.txt (verbalized triples via triple_to_text model)
- Experimental Layer (LM Integration)
– dataset_creation (create_interlm_data.py, weblg_generate_dataset.py)
– model_training (hs_detection.py, intermediate_pretraining.py, triple_to_text.py)
– Outputs: fine-tuned models (StereoKG-DT-SK, StereoKG-DT-UK on HuggingFace)
- External Services & Libraries
– Reddit API, Twitter Developer API
– OpenIE (for triple extraction)
– DistilBERT-CoLA, SimpleTransformers, HuggingFace Transformers, PyTorch, NLTK

##### Relationships and Data Flow
- Credentials.py feeds API keys and file paths into scrape_reddit.py and scrape_twitter.py.
- Raw text flows into data_processing scripts in sequence: conversion → clustering → triple extraction → triple selection.
- Selected triples are written to kg/stereoKG.tsv.
- triple_to_text (in both data_processing and lm_integration) converts triples to natural language, producing stereoKG_linearised.txt.
- The KG (structured & linearized) is consumed by lm_integration/dataset_creation to build training corpora.
- Datasets feed into model_training scripts, culminating in new model artifacts.

##### Architectural Patterns & Principles
- Pipeline/Batch Processing: clear stages from ingestion through transformation to output.
- Modular Scripts: each script handles one responsibility (single‐responsibility principle).
- Configuration Isolation: Credentials.py and questions folder strictly hold metadata and templates.
- Reusability: triple_to_text is used both for KG linearization and for LM experiment data preparation.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Configuration Layer"
        Cred["Credentials.py"]:::config
        Qs["questions/"]:::config
    end

    subgraph "Ingestion Layer"
        Reddit["scrape_reddit.py"]:::ingestion
        Twitter["scrape_twitter.py"]:::ingestion
    end

    subgraph "Transformation Layer"
        SM["StatementMaker.py"]:::processing
        Q2S["question_to_statement.py"]:::processing
        Inf["inflection.py"]:::processing
        FC["fast_clustering.py"]:::processing
        TG["triple_generation.py"]:::processing
        TS["triple_selection.py"]:::processing
        Utils["utils.py"]:::processing
        TT_transform["triple_to_text (Transformation)"]:::processing
    end

    subgraph "Storage Layer"
        KG["stereoKG.tsv"]:::storage
        KG_lin["stereoKG_linearised.txt"]:::storage
    end

    subgraph "Experimental Layer"
        CI["create_interlm_data.py"]:::experiment
        CW["weblg_generate_dataset.py"]:::experiment
        HS["hs_detection.py"]:::experiment
        IP["intermediate_pretraining.py"]:::experiment
        TT_exp["triple_to_text.py"]:::experiment
        Models["StereoKG-DT-SK / DT-UK"]:::experiment
    end

    subgraph "External Services"
        RedditAPI["Reddit API"]:::external
        TwitterAPI["Twitter API"]:::external
        OpenIE["OpenIE"]:::external
        Distil["DistilBERT-CoLA"]:::external
        Libraries["NLTK / PyTorch / Transformers"]:::external
    end

    Cred --> Reddit
    Cred --> Twitter

    RedditAPI -.-> Reddit
    TwitterAPI -.-> Twitter

    Reddit -->|"raw posts"| SM
    Twitter -->|"raw tweets"| SM

    SM --> Q2S --> Inf --> FC --> TG --> TS --> KG
    TG --> OpenIE
    TS -.-> Distil

    KG --> KG_lin
    TS -->|"selected triples"| TT_transform --> KG_lin

    KG_lin --> CI
    KG_lin --> CW

    CI --> HS
    CI --> IP
    CW --> IP

    IP --> TT_exp --> Models

    Libraries -.-> SM
    Libraries -.-> Q2S
    Libraries -.-> Inf
    Libraries -.-> FC
    Libraries -.-> TG
    Libraries -.-> TS
    Libraries -.-> TT_transform
    Libraries -.-> CI
    Libraries -.-> CW
    Libraries -.-> HS
    Libraries -.-> IP
    Libraries -.-> TT_exp

    click Cred "https://github.com/uds-lsv/stereokg/blob/main/Credentials.py"
    click Qs "https://github.com/uds-lsv/stereokg/tree/main/questions/"
    click Reddit "https://github.com/uds-lsv/stereokg/blob/main/data_extraction/scrape_reddit.py"
    click Twitter "https://github.com/uds-lsv/stereokg/blob/main/data_extraction/scrape_twitter.py"
    click SM "https://github.com/uds-lsv/stereokg/blob/main/data_processing/StatementMaker.py"
    click Q2S "https://github.com/uds-lsv/stereokg/blob/main/data_processing/question_to_statement.py"
    click Inf "https://github.com/uds-lsv/stereokg/blob/main/data_processing/inflection.py"
    click FC "https://github.com/uds-lsv/stereokg/blob/main/data_processing/fast_clustering.py"
    click TG "https://github.com/uds-lsv/stereokg/blob/main/data_processing/triple_generation.py"
    click TS "https://github.com/uds-lsv/stereokg/blob/main/data_processing/triple_selection.py"
    click Utils "https://github.com/uds-lsv/stereokg/blob/main/data_processing/utils.py"
    click KG "https://github.com/uds-lsv/stereokg/blob/main/kg/stereoKG.tsv"
    click KG_lin "https://github.com/uds-lsv/stereokg/blob/main/kg/stereoKG_linearised.txt"
    click CI "https://github.com/uds-lsv/stereokg/blob/main/lm_integration/dataset_creation/create_interlm_data.py"
    click CW "https://github.com/uds-lsv/stereokg/blob/main/lm_integration/dataset_creation/weblg_generate_dataset.py"
    click HS "https://github.com/uds-lsv/stereokg/blob/main/lm_integration/model_training/hs_detection.py"
    click IP "https://github.com/uds-lsv/stereokg/blob/main/lm_integration/model_training/intermediate_pretraining.py"
    click TT_exp "https://github.com/uds-lsv/stereokg/blob/main/lm_integration/model_training/triple_to_text.py"

    classDef config fill:#cccccc,stroke:#333,stroke-width:1px
    classDef ingestion fill:#a2cffe,stroke:#333,stroke-width:1px
    classDef processing fill:#a2fca2,stroke:#333,stroke-width:1px
    classDef storage fill:#fff2a2,stroke:#333,stroke-width:1px
    classDef experiment fill:#d5a2fc,stroke:#333,stroke-width:1px
    classDef external fill:#f0f0f0,stroke-dasharray:5 5,stroke:#333,stroke-width:1px
```
