# ananya-ayasi/gpu-distributed architecture

> Architecture diagram of the GitHub repository ananya-ayasi/gpu-distributed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ananya-ayasi/gpu-distributed
- Repository: https://github.com/ananya-ayasi/gpu-distributed

## Overview

##### Project type and purpose
- This is a hands-on PyTorch example repository for learning and demonstrating distributed (multi-GPU) model training and inference. It contains scripts, a config file, and a notebook to load data, define a GPT-2 style model, and orchestrate distributed training with PyTorch DDP.

##### File structure
- Top-level scripts:
- main.py – entry point that parses the YAML config and launches training
- trainer.py – encapsulates the training loop, batch distribution, and DDP setup
- model.py – defines the Transformer/GPT-2 style model architecture
- char_dataset.py – implements dataset creation and preprocessing from data/input.txt
- gpt2_train_cfg.yaml – training hyperparameters and DDP settings
- Distributed_Data_Parallel.ipynb – notebook demonstrating interactive usage
- Supporting files: data/input.txt as raw text corpus; requirements.txt for Python dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration Layer"
        CFG["gpt2_train_cfg.yaml"]:::config
    end

    subgraph "Data Layer"
        RAW["data/input.txt"]:::data
        DS["char_dataset.py"]:::code
    end

    subgraph "Model Layer"
        M["model.py"]:::code
    end

    subgraph "Training Layer"
        TR["trainer.py"]:::code
    end

    subgraph "Orchestration"
        MAIN["main.py"]:::code
    end

    subgraph "Distributed GPU Workers"
        direction TB
        subgraph DDP_Group
            W1["GPU Worker 1"]:::compute
            WN["GPU Worker N"]:::compute
        end
    end

    subgraph "Storage & Logging"
        CKPT["Checkpoints"]:::data
        LOGS["Logs"]:::data
    end

    NB["Distributed_Data_Parallel.ipynb"]:::code
    REQ["requirements.txt"]:::code
    RD["README.md"]:::code

    MAIN -->|"reads config"| CFG
    MAIN -->|"loads raw data"| RAW
    RAW --> DS
    MAIN -->|"creates Dataset instance"| DS
    MAIN -->|"creates Model instance"| M
    MAIN -->|"initializes Trainer"| TR
    TR -->|"spawns DDP processes"| W1
    W1 -->|"forward/backward & gradient sync"| WN
    W1 -->|"save checkpoint"| CKPT
    W1 -->|"write logs"| LOGS
    NB -->|"interactive orchestration"| MAIN

    click RAW "https://github.com/ananya-ayasi/gpu-distributed/blob/main/data/input.txt"
    click DS "https://github.com/ananya-ayasi/gpu-distributed/blob/main/char_dataset.py"
    click CFG "https://github.com/ananya-ayasi/gpu-distributed/blob/main/gpt2_train_cfg.yaml"
    click M "https://github.com/ananya-ayasi/gpu-distributed/blob/main/model.py"
    click TR "https://github.com/ananya-ayasi/gpu-distributed/blob/main/trainer.py"
    click MAIN "https://github.com/ananya-ayasi/gpu-distributed/blob/main/main.py"
    click NB "https://github.com/ananya-ayasi/gpu-distributed/blob/main/Distributed_Data_Parallel.ipynb"
    click REQ "https://github.com/ananya-ayasi/gpu-distributed/blob/main/requirements.txt"
    click RD "https://github.com/ananya-ayasi/gpu-distributed/blob/main/README.md"

    classDef code fill:#cce5ff,stroke:#00529B,color:#000
    classDef data fill:#d4edda,stroke:#40754c,color:#000
    classDef config fill:#fff3cd,stroke:#8a6d1d,color:#000
    classDef compute fill:#fde2b7,stroke:#d48f0b,color:#000
```
