How dinobby/magdi works
Project type and purpose
• This is a machine‐learning research codebase (MAGDi) for structured distillation of multi-agent interaction graphs to improve reasoning in smaller language models.
• It provides a data preparation pipeline, node‐embedding generation, model definition, training routine, and evaluation script.
File structure
• Top‐level scripts:
• data_utils.py – functions for parsing MAG JSONs and preparing dataset splits
• get_node_emb.py – script to compute or load initial node embeddings
• model.py – PyTorch/TensorFlow model definition of the student network augmented by MAG information
• train.py – training loop, hyperparameters, checkpoint saving
• test.py – evaluation loop, inference settings, metrics reporting
• utils.py – shared utilities (logging, argument parsing, metrics)
• Supplementary files: requirements.txt, LICENSE, README.md
• Data directories (referenced in README but not in the repo) – checkpoints/, MAG/, node_emb/, test_data/
Extract architecture insights from the README
• Training pipeline has four main phases:
1. MAG data preparation
2. Node embedding initialization
3. Student model training with distillation (MAGDi)
4. Model evaluation on held-out test sets
• External storage: Google Drive hosts large JSONs, embeddings, and checkpoints
Main components
a. Data Repository (external)
• MAG JSON files (ARC, ECQA, GSM8K, MATH, SQA)
• Test JSON files
b. Data Preparation Module (data_utils.py)
• Reads MAG JSON → builds graph structures → outputs batched examples
c. Embedding Generation Module (get_node_emb.py)
• Takes reasoning sequences → average pooling → serializes node_emb.pkl
d. Student Model Module (model.py)
• Defines the neural network augmented with graph embeddings
e. Training Engine (train.py)
• Pulls preprocessed data + initial embeddings → performs forward/backward passes → writes checkpoints/ logs
f. Evaluation Engine (test.py)
• Loads checkpoints + test data → runs inference → computes accuracy / reasoning metrics
g. Utility Layer (utils.py)
• Shared logging, argument parsing, metric functions
Relationships & data flows
• Training Engine → Checkpoint Store (on local disk or Google Drive)
Architectural patterns & design principles
• Pipeline Architecture: discrete, sequential phases
• Modular Scripts: each script is a pluggable component (prep, embed, train, test)
• Separation of Concerns: data I/O vs. model definition vs. training logic vs. evaluation
• Externalization of large artifacts (data, embeddings, checkpoints) to Google Drive
Technologies & frameworks
• Python 3.10
• PyTorch (or TensorFlow) for model.py, train.py, test.py
• JSON for MAG and test data
• Pickle for serialized node embeddings
• pip/requirements.txt for dependency management
Detail & thoroughness
• Read/write interactions with every directory (MAG, node_emb, checkpoints, test_data)
• Hyperparameter flow into train.py (e.g., lr, batch_size)