How twitter-research/tgn works
Project Type and Purpose
• Project Type: It is a research-driven deep learning framework focused on dynamic (temporal) graphs.
• Purpose: To efficiently model time-evolving graphs using advanced techniques like memory modules, temporal attention, and graph operators. The design needs to support experiments with different configurations (e.g., baselines like Jodie and DyRep) and both training paradigms (self-supervised and supervised).
File Structure
• Top-Level Files and Directories:
• Training Scripts: train_self_supervised.py and train_supervised.py – These serve as entry points.
• Model Directory: Contains files like tgn.py (main model), temporal_attention.py, time_encoding.py – depicting the core deep learning model.
• Modules Directory: Includes components for embedding, memory, memory updater, message aggregator, and message function. These are building blocks of the TGN architecture.
• Evaluation Directory: Contains evaluation/evaluation.py – used for model performance evaluation.
• Utils Directory: Holds data processing and preprocessing scripts to transform raw datasets into a usable format.
Key Components
a. Data and Preprocessing:
• External Data Sources: CSV files from datasets like Wikipedia and Reddit (stored in a data folder as per the README).
• Preprocessing Module: utils/preprocess_data.py and utils/data_processing.py.
b. Training Modules:
• Self-Supervised and Supervised Training Pipelines: Represented by train_self_supervised.py and train_supervised.py which pick up the processed data, set up experiments, and invoke the model.
• Hyperparameter and configuration management (command-line flags) play a role in steering experiments.
c. Core TGN Model Architecture:
• Main Model: model/tgn.py, which orchestrates how a temporal graph is processed.
• Subcomponents Inside the Model:
• Temporal Attention mechanism: model/temporal_attention.py.
• Time Encoding: model/time_encoding.py.
• Memory Module and its updater: modules/memory.py and modules/memory_updater.py.
• Message Aggregation and Message Functions: modules/message_aggregator.py and modules/message_function.py.
• Embedding Module: modules/embedding_module.py.
d. Evaluation Component:
• Evaluation script: evaluation/evaluation.py, which likely computes performance metrics and provides feedback.
Relationships and Interactions
• Data flows from the external data source to the Preprocessing module.
• The Preprocessing module converts raw data to a format acceptable by the training scripts.
• The Training pipeline then loads the preprocessed data, instantiates the TGN model (with its submodules), and initiates the learning process.
• The TGN model itself is built from several interdependent components (e.g., it uses temporal attention to weigh dynamic relationships and a memory module to store evolving node states).
• Evaluation scripts measure the performance, possibly feeding back into training (especially for experiments like ablation studies).
Architectural Patterns and Design Principles
• Modularity: Clearly separated directories for models, modules, and utilities show that each component is designed to be interchangeable (for example, swapping out the embedding or memory modules).
• Separation of Concerns: Preprocessing, training mechanisms, model architecture, and evaluation are all decoupled.
• Extensibility: With multiple options for message functions, aggregator types, or memory updaters, the design supports easy experimentation with alternatives (e.g., using different baselines like Jodie or DyRep).