# dinobby/magdi architecture

> Architecture diagram of the GitHub repository dinobby/magdi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dinobby/magdi
- Repository: https://github.com/dinobby/magdi

## Overview

##### Project type and purpose
- This is a machine‐learning research codebase (MAGDi) for structured distillation of multi-agent interaction graphs to improve reasoning in smaller language models.
- It provides a data preparation pipeline, node‐embedding generation, model definition, training routine, and evaluation script.

##### File structure
- Top‐level scripts:
- data_utils.py – functions for parsing MAG JSONs and preparing dataset splits
- get_node_emb.py – script to compute or load initial node embeddings
- model.py – PyTorch/TensorFlow model definition of the student network augmented by MAG information
- train.py – training loop, hyperparameters, checkpoint saving
- test.py – evaluation loop, inference settings, metrics reporting
- utils.py – shared utilities (logging, argument parsing, metrics)
- Supplementary files: requirements.txt, LICENSE, README.md
- Data directories (referenced in README but not in the repo) – checkpoints/, MAG/, node_emb/, test_data/

##### Extract architecture insights from the README
- Training pipeline has four main phases:
1. MAG data preparation
2. Node embedding initialization
3. Student model training with distillation (MAGDi)
4. Model evaluation on held-out test sets
- External storage: Google Drive hosts large JSONs, embeddings, and checkpoints

##### Main components
a. Data Repository (external)
- MAG JSON files (ARC, ECQA, GSM8K, MATH, SQA)
- Test JSON files
b. Data Preparation Module (data_utils.py)
- Reads MAG JSON → builds graph structures → outputs batched examples
c. Embedding Generation Module (get_node_emb.py)
- Takes reasoning sequences → average pooling → serializes node_emb.pkl
d. Student Model Module (model.py)
- Defines the neural network augmented with graph embeddings
e. Training Engine (train.py)
- Pulls preprocessed data + initial embeddings → performs forward/backward passes → writes checkpoints/ logs
f. Evaluation Engine (test.py)
- Loads checkpoints + test data → runs inference → computes accuracy / reasoning metrics
g. Utility Layer (utils.py)
- Shared logging, argument parsing, metric functions

##### Relationships & data flows
- Training Engine → Checkpoint Store (on local disk or Google Drive)

##### Architectural patterns & design principles
- Pipeline Architecture: discrete, sequential phases
- Modular Scripts: each script is a pluggable component (prep, embed, train, test)
- Separation of Concerns: data I/O vs. model definition vs. training logic vs. evaluation
- Externalization of large artifacts (data, embeddings, checkpoints) to Google Drive

##### Technologies & frameworks
- Python 3.10
- PyTorch (or TensorFlow) for model.py, train.py, test.py
- JSON for MAG and test data
- Pickle for serialized node embeddings
- pip/requirements.txt for dependency management

##### Detail & thoroughness
- Read/write interactions with every directory (MAG, node_emb, checkpoints, test_data)
- Hyperparameter flow into train.py (e.g., lr, batch_size)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Repository"
        direction TB
        MAGs[(MAG JSON Files)]:::datastore
        Tests[(Test JSON Files)]:::datastore
    end

    GoogleDrive[(Google Drive)]:::external

    MAGs -->|"MAG JSONs"| DataPrep
    GoogleDrive --> MAGs
    GoogleDrive --> nodeEmb
    GoogleDrive --> Checkpoints

    DataPrep["Data Preparation\n(data_utils.py)"]:::processing
    click DataPrep "https://github.com/dinobby/magdi/blob/main/data_utils.py"

    DataPrep -->|"Graph batches"| EmbedGen

    EmbedGen["Embedding Generation\n(get_node_emb.py)"]:::processing
    click EmbedGen "https://github.com/dinobby/magdi/blob/main/get_node_emb.py"

    EmbedGen -->|"node_emb.pkl"| nodeEmb[(node_emb.pkl)]:::datastore

    nodeEmb -->|"initial embeddings"| Train

    DataPrep -->|"training data"| Train

    Model["Student Model\n(model.py)"]:::model
    click Model "https://github.com/dinobby/magdi/blob/main/model.py"
    Train["Training Engine\n(train.py)"]:::model
    click Train "https://github.com/dinobby/magdi/blob/main/train.py"

    Train -->|"checkpoints"| Checkpoints[(Checkpoints)]:::datastore

    Tests -->|"test data"| Eval

    Eval["Evaluation Engine\n(test.py)"]:::model
    click Eval "https://github.com/dinobby/magdi/blob/main/test.py"

    Checkpoints -->|"load checkpoints"| Eval

    Eval -->|"metrics output"| Metrics["Metrics Output"]:::processing

    Utils["Utilities\n(utils.py)"]:::utility
    click Utils "https://github.com/dinobby/magdi/blob/main/utils.py"

    Utils -.-> DataPrep
    Utils -.-> EmbedGen
    Utils -.-> Train
    Utils -.-> Eval

    Req["requirements.txt"]:::doc
    click Req "https://github.com/dinobby/magdi/blob/main/requirements.txt"
    Readme["README.md"]:::doc
    click Readme "https://github.com/dinobby/magdi/blob/main/README.md"
    License["LICENSE"]:::doc
    click License "https://github.com/dinobby/magdi/tree/main/LICENSE"

    classDef processing fill:#a2f5a2,stroke:#333,stroke-width:1px;
    classDef datastore fill:#a2c4f5,stroke:#333,stroke-width:1px;
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px;
    classDef model fill:#f5cba2,stroke:#333,stroke-width:1px;
    classDef utility fill:#cccccc,stroke:#333,stroke-width:1px;
    classDef doc fill:#f5f5b5,stroke:#333,stroke-width:1px;
```
