# kilab-lyu/graphprobe architecture

> Architecture diagram of the GitHub repository kilab-lyu/graphprobe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kilab-lyu/graphprobe
- Repository: https://github.com/kilab-lyu/graphprobe

## Overview

##### Project Type and Purpose
- This is an open-source research toolkit for “Knowledge Probing for Graph Representation Learning.”
- It provides a model-agnostic pipeline that:
a. Trains a variety of graph embedding (GE) models on different graph datasets.
b. Stores learned embeddings.
c. Applies multiple probing tasks (CentProbe, DistProbe, etc.) to evaluate how well embeddings capture structural/semantic knowledge.
d. Produces evaluation results and visualizations.

##### High-Level File Structure Analysis
- Top-level orchestrators:
- main.py: entry point that reads configuration, loops over GE models and probing tasks.
- GE_train.py & GE_model.py: handle graph embedding training.
- DownstreamTask.py: dispatches and runs probe tasks.
- Data layer:
- Datasets.py + Utility/data/: dataset loaders, samplers.
- Cora&FlickrInfo.py, ProteinsDataset.py, MutagCos.py, ppi.py, yelp.py, amazon_book.py, etc.
- Model layer:
- Utility/model/GCL, Utility/model/base, Utility/model/general_recommender: implementations of GraphCL, GCN, GAT, GraphSAGE, Node2Vec, etc.
- Probing layer:
- probe/: various probing modules (CentProbe, DistProbe, MIProbe, etc.).
- Configuration & Utilities:
- Utility/constant.py: all hyperparameters & datasets/models lists.
- Utility/parser.py, Utility/helper.py, Utility/metrics.py, Utility/utils.py.
- Utility/reckit/: Cython-accelerated evaluation metrics, logging, plotting.
- Output:
- Embedding files under EMB_ROOT (e.g. './Embedding').
- results/: text files of probe evaluations per dataset/model.
- drawRadar.py, DrawLinePic.py, img/: scripts & images for visualization.

##### Key Insights from README
- The pipeline is driven entirely by constants in Utility/constant.py (model list, dataset list, downstream tasks).
- The code supports both transductive and inductive modes.
- Probing tasks selection is done via Dict_probe_emb mapping to List_permit_args.

##### Components
a. Main Components
1. Configuration Manager
- Reads Utility/constant.py
- Exposes: List_GE_Model, List_permit_args, Dict_probe_emb, device, modes
2. Data Ingestion & Preprocessing
- Datasets.py + Utility/data loaders & samplers
- Specialized dataset scripts (cora.py, flickr.py, etc.)
3. Graph Embedding Training
- GE_train.py orchestrates training loops
- Graph model implementations in Utility/model/… and GE_model.py
- Uses PyTorch Geometric, DGL, torch-cluster, Cython (Utility/reckit)
- Outputs serialized embeddings to EMB_ROOT
4. Knowledge Probing
- DownstreamTask.py calls specific probe modules under probe/
- Probes consume embeddings + dataset metadata
- Evaluation metrics via Utility/reckit (Cython-backed) and Utility/metrics
- Writes results under results/…
5. Visualization
- DrawLinePic.py, drawRadar.py
- img/ for static assets
6. Utilities & Shared Services
- Logging (Utility/reckit/logger.py)
- Common helper functions (Utility/helper.py, Utility/utils.py)
- Augmentors & losses for contrastive learning (Utility/model/GCL/Utility/…)

b. Relationships & Data Flow
1. Configuration Manager feeds parameters → Data Ingestion
2. Data Ingestion → Graph Embedding Trainer
3. Trainer → Serialized embeddings
4. Serialized embeddings + Configuration → Probing Tasks
5. Probing Tasks → Evaluation Results
6. Results → Visualization scripts

c. Architectural Patterns & Principles
- Modular, plugin-like design: new GE models in Utility/model, new probes under probe/.
- Separation of concerns: configuration, data, models, probes, evaluation, visualization all in distinct modules.
- Performance optimization: Cython modules under Utility/reckit/cython.
- Model-agnostic pipeline: same orchestration for any GE model.

d. Key Technologies & Frameworks
- Python 3.x, PyTorch 2.1+cu118, PyTorch-Geometric, DGL, Torch-Cluster/Scatter/Sparse, Torchaudio, Torchvision
- Cython for critical loops (Utility/reckit/cython)
- Numpy, Matplotlib or similar via DrawLinePic.py

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration & Utilities
    subgraph "Configuration & Utilities"
        direction TB
        ConfigManager["Configuration Manager"]:::config
        Utilities["Utilities & Shared Services"]:::config
    end

    %% Data & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        direction TB
        DataLoader["Core Loader & Sampler"]:::data
        SpecializedDatasets["Specialized Dataset Scripts"]:::data
    end

    %% Model Training
    subgraph "Graph Embedding Training"
        direction TB
        TrainerOrch["Training Orchestrator\n(GE_train.py, GE_model.py)"]:::train
        BaseInterfaces["Base Interfaces\n(abstract_recommender.py,\nconfigurator.py)"]:::train
        RecommenderModels["General & Contrastive Models\n(SGL.py, GraphCL.py,\nnode2vec.py, randomwalk.py)"]:::train
        GCLModule["GCL Module & Utilities"]:::train
    end

    %% Probing & Evaluation
    subgraph "Knowledge Probing"
        direction TB
        ProbeDispatcher["Probe Dispatcher\n(DownstreamTask.py)"]:::probe
        ProbeImpl["Probe Implementations\n(CentProbe, DistProbe,\nMIProbe, etc.)"]:::probe
    end

    %% Outputs & Visualization
    subgraph "Outputs & Visualization"
        direction TB
        EMB["EMB_ROOT\n(Serialized Embeddings)"]:::train
        Results["results/"]:::probe
        Visualization["Visualization\n(DrawLinePic.py,\ndrawRadar.py)"]:::viz
        Images["img/"]:::viz
    end

    %% Flows
    ConfigManager -->|reads config| DataLoader
    ConfigManager -->|feeds parameters| TrainerOrch
    ConfigManager -->|provides task list| ProbeDispatcher
    Utilities -->|helpers & metrics| TrainerOrch
    Utilities -->|Cython eval & logging| ProbeDispatcher

    DataLoader --> SpecializedDatasets
    DataLoader --> TrainerOrch

    TrainerOrch --> BaseInterfaces
    BaseInterfaces --> RecommenderModels
    BaseInterfaces --> GCLModule
    RecommenderModels --> EMB
    GCLModule --> EMB

    EMB --> ProbeDispatcher
    ProbeDispatcher --> ProbeImpl
    ProbeImpl --> Results

    Results --> Visualization
    Visualization --> Images

    %% Click Events
    click ConfigManager "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/constant.py"
    click ConfigManager "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/parser.py"
    click ConfigManager "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/reckit/configurator.py"
    click Utilities "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/helper.py"
    click Utilities "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/utils.py"
    click Utilities "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/metrics.py"
    click Utilities "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/reckit/logger.py"
    click Utilities "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/reckit/evaluator.py"
    click Utilities "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/reckit/plot/style.py"
    click Utilities "https://github.com/kilab-lyu/graphprobe/tree/main/Utility/reckit/cython/"
    click DataLoader "https://github.com/kilab-lyu/graphprobe/blob/main/Datasets.py"
    click DataLoader "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/data/dataset.py"
    click DataLoader "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/data/sampler.py"
    click SpecializedDatasets "https://github.com/kilab-lyu/graphprobe/blob/main/Cora&FlickrInfo.py"
    click SpecializedDatasets "https://github.com/kilab-lyu/graphprobe/blob/main/ProteinsDataset.py"
    click SpecializedDatasets "https://github.com/kilab-lyu/graphprobe/blob/main/MutagCos.py"
    click SpecializedDatasets "https://github.com/kilab-lyu/graphprobe/blob/main/ppi.py"
    click SpecializedDatasets "https://github.com/kilab-lyu/graphprobe/blob/main/yelp.py"
    click SpecializedDatasets "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/data/amazon_book.py"
    click TrainerOrch "https://github.com/kilab-lyu/graphprobe/blob/main/GE_train.py"
    click TrainerOrch "https://github.com/kilab-lyu/graphprobe/blob/main/GE_model.py"
    click BaseInterfaces "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/model/base/abstract_recommender.py"
    click BaseInterfaces "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/model/base/configurator.py"
    click RecommenderModels "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/model/general_recommender/SGL.py"
    click RecommenderModels "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/GraphCL.py"
    click RecommenderModels "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/GraphRNN.py"
    click RecommenderModels "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/SGL.py"
    click RecommenderModels "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/node2vec.py"
    click RecommenderModels "https://github.com/kilab-lyu/graphprobe/blob/main/Utility/randomwalk.py"
    click GCLModule "https://github.com/kilab-lyu/graphprobe/tree/main/Utility/model/GCL"
    click ProbeDispatcher "https://github.com/kilab-lyu/graphprobe/blob/main/DownstreamTask.py"
    click ProbeImpl "https://github.com/kilab-lyu/graphprobe/blob/main/probe/CentProbe.py"
    click ProbeImpl "https://github.com/kilab-lyu/graphprobe/blob/main/probe/DistProbe.py"
    click ProbeImpl "https://github.com/kilab-lyu/graphprobe/blob/main/probe/MIProbe.py"
    click ProbeImpl "https://github.com/kilab-lyu/graphprobe/blob/main/probe/CategoryProbe.py"
    click ProbeImpl "https://github.com/kilab-lyu/graphprobe/blob/main/probe/ClusterProbe.py"
    click ProbeImpl "https://github.com/kilab-lyu/graphprobe/blob/main/probe/GstructProbe.py"
    click Visualization "https://github.com/kilab-lyu/graphprobe/blob/main/DrawLinePic.py"
    click Visualization "https://github.com/kilab-lyu/graphprobe/blob/main/drawRadar.py"

    %% Styles
    classDef config fill:#D6EAF8,stroke:#1B4F72,color:#1B2631
    classDef data fill:#D5F5E3,stroke:#145A32,color:#0B5345
    classDef train fill:#FADBD8,stroke:#922B21,color:#641E16
    classDef probe fill:#E8DAEF,stroke:#6C3483,color:#4A235A
    classDef viz fill:#E5E7E9,stroke:#424949,color:#1C2833
```
