# yuxiawu/simpledyg architecture

> Architecture diagram of the GitHub repository yuxiawu/simpledyg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yuxiawu/simpledyg
- Repository: https://github.com/yuxiawu/simpledyg

## Overview

##### Project type and purpose
- This is a research‐focused ML pipeline (“SimpleDyG”) implementing a Transformer for dynamic graph modeling.
- It includes data preprocessing, training, tokenization/vocabulary generation, and evaluation.

##### High-level components
a. Data Ingestion & Preprocessing
- all_data/…: raw datasets (dialog, hepth, movie, uci) and per‐dataset scripts (preprocess.py, data_trans.py, train_word2vec.py).
- csv2res.sh & csv2resources.py: transforms CSV+NPY into the final format under resources/.

b. Data Store (Preprocessed Artifacts)
- resources/: final train/val/test link‐prediction files for each dataset & timestamp.
- tokenizers/: HuggingFace tokenizer JSONs & configs, per dataset/timestamp.
- vocabs/: vocabulary JSONs matching each tokenizer.

c. Code Modules
- data/: Dataset and language_model wrapper (data ingestion API).
- models/: Transformer model definitions (configuration_gpt2.py, modeling_gpt2.py, utilities).
- utils/: argument parsers, evaluation routines (Evaluation.py), language_model helper, generic model helper.

d. Training & Orchestration
- main.py: entry point tying data, model, training loop, logging to runs/.
- train_seed_*.sh: shell wrappers invoking main.py with dataset‐specific arguments.
- requirements.txt: Python dependencies.

e. Output & Evaluation
- During training:
– tokenizers/, vocabs/, output/ (checkpoints), runs/ (TensorBoard).
- eval_single_step.py: runs one‐step evaluation, writing metrics to results/.

##### Relationships & Data Flow
1) Raw data in all_data/ → run each all_data/*/preprocess.py → CSV + NPY outputs (ml_*.csv/.npy).
2) csv2res.sh → csv2resources.py → write standardized link‐prediction files to resources/.
3) Training scripts (train_seed_*.sh → main.py):
– main.py loads resources/, instantiates a tokenizer (writes to tokenizers/), builds vocab (vocabs/).
– Data module loads examples, DataLoader → utils/model.py orchestrates training loop.
– models/modeling_gpt2.py defines the Transformer, configuration_utils.py sets hyperparams.
– Checkpoints saved to output/, logs to runs/.
4) Evaluation: eval_single_step.py loads checkpoint from output/, loads resources/ for test split, runs utils/Evaluation.py → results/.

##### Architectural patterns & design principles
- Pipeline architecture: clear segmentation into Preprocessing → Data Storage → Model Training → Evaluation.
- Modular codebase:
- data/ encapsulates dataset handling
- models/ encapsulates all modeling logic
- utils/ holds cross‐cutting concerns (parsing args, evaluation)
- Reproducibility & logging: shell scripts + main.py + tensorboard logs ensure experiment tracking.

## Diagram (Mermaid)

```mermaid
flowchart TD
    RD["Raw Data\n(all_data/)"]:::storage
    PP["Preprocessing Scripts"]:::compute
    CSV["CSV2RES Converter"]:::compute
    RES["Resources Store\n(resources/)"]:::storage
    TV["Tokenizer & Vocab Generation\n(tokenizers/, vocabs/)"]:::artifact
    DL["Data Loader Module\ndata/"]:::library
    MT["Model Trainer\n(main.py, models/, utils/)"]:::compute
    CS["Checkpoint Store\n(output/)"]:::artifact
    EM["Evaluation Module"]:::compute
    RS["Results\n(results/)"]:::artifact

    RD --> PP
    PP --> CSV
    CSV --> RES
    RES --> TV
    TV --> DL
    DL --> MT
    MT --> CS
    CS --> EM
    EM --> RS

    click RD "https://github.com/yuxiawu/simpledyg/tree/main/all_data/"
    click PP "https://github.com/yuxiawu/simpledyg/tree/main/all_data/"
    click CSV "https://github.com/yuxiawu/simpledyg/blob/main/csv2res.sh"
    click CSV "https://github.com/yuxiawu/simpledyg/blob/main/csv2resources.py"
    click RES "https://github.com/yuxiawu/simpledyg/tree/main/resources/"
    click TV "https://github.com/yuxiawu/simpledyg/tree/main/tokenizers/"
    click TV "https://github.com/yuxiawu/simpledyg/tree/main/vocabs/"
    click DL "https://github.com/yuxiawu/simpledyg/tree/main/data/"
    click DL "https://github.com/yuxiawu/simpledyg/blob/main/data/__init__.py"
    click DL "https://github.com/yuxiawu/simpledyg/blob/main/data/dataset/language_model.py"
    click MT "https://github.com/yuxiawu/simpledyg/blob/main/main.py"
    click MT "https://github.com/yuxiawu/simpledyg/blob/main/models/configuration_gpt2.py"
    click MT "https://github.com/yuxiawu/simpledyg/blob/main/models/configuration_utils.py"
    click MT "https://github.com/yuxiawu/simpledyg/blob/main/models/modeling_gpt2.py"
    click MT "https://github.com/yuxiawu/simpledyg/blob/main/models/modeling_utils.py"
    click MT "https://github.com/yuxiawu/simpledyg/blob/main/models/file_utils.py"
    click MT "https://github.com/yuxiawu/simpledyg/tree/main/models/"
    click MT "https://github.com/yuxiawu/simpledyg/tree/main/utils/"
    click CS "https://github.com/yuxiawu/simpledyg/tree/main/output/"
    click EM "https://github.com/yuxiawu/simpledyg/blob/main/eval_single_step.py"
    click EM "https://github.com/yuxiawu/simpledyg/blob/main/utils/Evaluation.py"
    click EM "https://github.com/yuxiawu/simpledyg/tree/main/results/"

    classDef storage fill:#a6cee3,stroke:#333,stroke-width:1px;
    classDef compute fill:#b2df8a,stroke:#333,stroke-width:1px;
    classDef library fill:#ffff99,stroke:#333,stroke-width:1px;
    classDef artifact fill:#fb9a99,stroke:#333,stroke-width:1px;
```
