# hkproj/pytorch-transformer-distributed architecture

> Architecture diagram of the GitHub repository hkproj/pytorch-transformer-distributed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hkproj/pytorch-transformer-distributed
- Repository: https://github.com/hkproj/pytorch-transformer-distributed

## Overview

##### Project Type and Purpose
- The project is centered around distributed training of an attention model using PyTorch. It is a research/ML training framework intended to allow for distributed execution (using torchrun with specified rendezvous settings) on multiple GPU nodes.
- The README clearly outlines both local and distributed training setups, including hardware setups (multiple nodes on Paperspace, network drive for saving checkpoints) and the integration with an external monitoring service (Weights & Biases).

##### File Structure
- Top-level files:
- Configuration and code files: config.py, dataset.py, model.py, train.py. These encapsulate the primary components of the training pipeline — model definition, dataset handling, training routines, and configurable parameters.
- requirements.txt and conda.txt: These define the Python environment and dependencies.
- .gitignore and README.md: Standard repository files.
- The “notes” directory includes supportive documents and examples that elaborate on distributed training and underlying gradient computations. This indicates that the project design may incorporate not only the production training pipeline but also supporting educational or experimental material.

##### Main Components
a. Distributed Training Environment:
- Multiple GPU nodes (slave nodes and a master node) that will run instances of train.py via torchrun. Each node runs parallel processes (using --nproc_per_node).
- The nodes communicate using a rendezvous backend (c10d), which should be shown as a central coordination point.
b. Training Pipeline:
- Core training components:
- config.py: Responsible for passing configuration settings.
- dataset.py: Handles data input and pre-processing.
- model.py: Defines the attention model to be trained.
- train.py: Orchestrates the training loop and integrates the above components.
c. External Services:
- Network drive: The training checkpoints and model weights are stored on a mounted network drive.
- Weights & Biases (W&B): Monitoring of training progress and logging.
d. Supporting/Development Materials:
- The “notes” directory provides example codes and additional explanations (like the ddp_template and gradient computation examples), which while not part of the production pipeline, give context to the design choices and debugging processes.

##### Relationships and Interactions
- Data Flow: The dataset (from dataset.py) feeds the model (model.py) through the training process in train.py.
- Configuration: config.py feeds both dataset and model settings.
- Distributed Communication: The training nodes are interconnected through the torch.distributed (c10d) backend, with the master node orchestrating the updates.
- External Interaction: The training nodes write checkpoints to the mounted network drive and send logs to W&B.
- Dependencies: Highlight that the training environment is built on PyTorch and utilizes distributed computing libraries and OS-level setups (e.g., the network setup instructions in the README).

##### Architectural Patterns and Design Principles
- A modular design where configuration, data loading, model creation, and training logic are separated into individual files.
- The design supports a distributed/multi-node architecture for high performance on multi-GPU setups, following a loose microservice-like approach where distinct components have clear responsibilities.

##### Specific Elements
- "GPU Node (Master)" and "GPU Node (Slave)" to represent the distributed environment.
- "Configuration Module (config.py)", "Dataset Handler (dataset.py)", "Model Constructor (model.py)", and "Training Orchestrator (train.py)".
- "Network Drive (Checkpoint Storage)" and "Weights & Biases (Monitoring)".
- The flow of configuration and data from config.py to the dataset and model.
- The parallel execution on multiple GPU nodes with synchronization via torch.distributed.
- The update flow from nodes to external storage and monitoring.
- Optionally, include annotations explaining that the setup is highly reliant on the Paperspace configuration and that the training environment utilizes a pre-defined rendezvous protocol for distributed training.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Training Pipeline"
        config["Configuration Module (config.py)"]:::module
        dataset["Dataset Handler (dataset.py)"]:::module
        model["Model Constructor (model.py)"]:::module
        train["Training Orchestrator (train.py)"]:::module
        
        config -->|"sendsConfig"| dataset
        config -->|"sendsConfig"| model
        dataset -->|"feedsData"| train
        model -->|"providesModel"| train
        config -->|"configData"| train
    end

    subgraph "Distributed Training Environment"
        master["GPU Node (Master)"]:::compute
        slave["GPU Node (Slave)"]:::compute
        coordination["Distributed Coordination (c10d Rendezvous)"]:::compute
        
        train -->|"executesOn"| master
        train -->|"executesOn"| slave
        master -->|"syncsVia"| coordination
        slave -->|"syncsVia"| coordination
    end

    subgraph "External Services"
        network["Network Drive (Checkpoint Storage)"]:::external
        wandb["Weights & Biases (Monitoring)"]:::external
        
        master -->|"savesCheckpoint"| network
        slave -->|"savesCheckpoint"| network
        master -->|"logsTo"| wandb
        slave -->|"logsTo"| wandb
    end

    subgraph "Supporting/Development Materials"
        notes["Supporting/Development Materials\n(notes/ & examples)"]:::module
    end

    train ---|"referenceExamples"| notes

    classDef module fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef compute fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef external fill:#F9E79F,stroke:#B7950B,stroke-width:2px;

    click config "https://github.com/hkproj/pytorch-transformer-distributed/blob/main/config.py"
    click dataset "https://github.com/hkproj/pytorch-transformer-distributed/blob/main/dataset.py"
    click model "https://github.com/hkproj/pytorch-transformer-distributed/blob/main/model.py"
    click train "https://github.com/hkproj/pytorch-transformer-distributed/blob/main/train.py"
    click notes "https://github.com/hkproj/pytorch-transformer-distributed/tree/main/notes/"
```
