# ghelanikirtan/transformer-using-pytorch-from-scratch architecture

> Architecture diagram of the GitHub repository ghelanikirtan/transformer-using-pytorch-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ghelanikirtan/transformer-using-pytorch-from-scratch
- Repository: https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch

## Overview

##### Project type and purpose
- This is a research-style, end-to-end Transformer language model implementation in PyTorch (“decoder-only”, GPT-style).
- It provides scripts for data preprocessing, vocabulary building, training, validation (perplexity), and single-prompt or interactive text generation.

##### File structure
- Top-level files:
- config.yaml (hyperparameters, file paths)
- requirements.txt (Python dependencies)
- training.log (runtime logging)
- Data directories (mentioned in README):
- data/processed/ (preprocessed token files, vocab.pt)
- checkpoints/ (model checkpoints)
- src/ directory:
- preprocess_data.py (reads raw text → tokenizes → builds vocab.pt)
- dataset.py (loads processed tokens + vocab → PyTorch Dataset/ DataLoader)
- network.py (defines Transformer model: embeddings, multi-head attention, feed-forward, layernorm)
- train.py (training loop: forward, loss, backward, optimizer, checkpointing, log)
- validate.py (evaluates on held-out data to compute perplexity)
- predict.py (generates text from a single prompt)
- run_lm.py (interactive console loop for generation)
- utils.py (configuration loader, logger setup, any other helpers)
- constants.py (device selection, seed setting)

##### README for additional insights
- Confirms architecture (scaled dot-product attention, multi-head, positional encodings, etc.)
- Mentions goals (train on larger corpora, add sampling strategies).

##### Detailed steps for the principal software engineer
- Left lane: “Data & Config” (config.yaml, raw data, preprocess)
- Middle lane: “Core Pipeline” (dataset.py → network.py → train.py/validate.py)
- Right lane: “Persistence & Inference” (checkpoints/, predict.py, run_lm.py)
4. Flow: config → preprocessing → data store → dataset → model → training → checkpoints + logs → validation → inference modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Config"
        ConfigLoader["Config Loader\n(config.yaml)"]:::config
        Preprocess["Preprocess Data"]:::compute
        DataStore["Processed Data & Vocab"]:::data
    end

    subgraph "Core Pipeline"
        Dataset["Dataset & DataLoader"]:::compute
        Network["Transformer Network"]:::compute
        Train["Training Loop"]:::compute
        Validate["Validation Module"]:::compute
    end

    subgraph "Persistence & Inference"
        Checkpoint["Checkpoints Store"]:::data
        Log["Training Log"]:::config
        Predict["Prediction Module\n(predict.py)"]:::compute
        RunLM["Interactive Inference\n(run_lm.py)"]:::compute
    end

    Utils["Utilities & Constants"]:::util

    %% Data & Config to Core Pipeline
    ConfigLoader -->|reads config| Preprocess
    ConfigLoader -->|reads config| Train
    ConfigLoader -->|reads config| Validate
    ConfigLoader -->|reads config| Predict
    ConfigLoader -->|reads config| RunLM

    Preprocess -->|tokenized dataset| DataStore
    DataStore -->|vocab + tokens| Dataset

    Dataset -->|batches| Train
    Network -->|model definition| Train
    Train -->|model state| Checkpoint
    Train -->|metrics| Log

    Validate -->|metrics| Log

    %% Inference path
    Checkpoint -->|load state| Predict
    DataStore -->|vocab| Predict

    Checkpoint -->|load state| RunLM
    DataStore -->|vocab| RunLM

    %% Utilities dotted connections
    Utils -.-> Preprocess
    Utils -.-> Dataset
    Utils -.-> Network
    Utils -.-> Train
    Utils -.-> Validate
    Utils -.-> Predict
    Utils -.-> RunLM

    %% Click Events
    click ConfigLoader "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/config.yaml"
    click Preprocess "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/preprocess_data.py"
    click Dataset "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/dataset.py"
    click Network "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/network.py"
    click Train "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/train.py"
    click Validate "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/validate.py"
    click Predict "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/predict.py"
    click RunLM "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/run_lm.py"
    click Utils "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/src/utils.py"
    click Checkpoint "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/tree/main/checkpoints/"
    click Log "https://github.com/ghelanikirtan/transformer-using-pytorch-from-scratch/blob/main/training.log"

    %% Legend
    subgraph Legend
        ComputeModule["Compute Module"]:::compute
        DataStoreLeg["Data Store"]:::data
        ConfigLeg["Config / Log"]:::config
        UtilLeg["Utilities"]:::util
    end

    %% Styles
    classDef compute fill:#ADD8E6,stroke:#000
    classDef data fill:#90EE90,stroke:#000,stroke-dasharray: 5 5
    classDef config fill:#FFFFE0,stroke:#000
    classDef util fill:#FFD1DC,stroke:#000,stroke-dasharray: 2 2
```
