# ayushsyntax/gpt-from-scratch-in-pytorch architecture

> Architecture diagram of the GitHub repository ayushsyntax/gpt-from-scratch-in-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayushsyntax/gpt-from-scratch-in-pytorch
- Repository: https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch

## Overview

##### Project Type and Purpose
- GPT-Mini is an open-source, research-style machine learning project: a compact, decoder-only Transformer language model implemented in PyTorch, trained on Shakespeare’s complete works for character-level text generation. It also includes scripts for training, evaluation, and a Gradio-based deployment.

##### File Structure Analysis
Top-level:
- configs/        – YAML ﬁles for model/training hyperparameters
- data/           – raw text corpus (tinyshakespeare.txt)
- deploy/         – inference app (Gradio via app.py)
- experiments/    – experiment-tracking notes or scripts
- src/            – core code
- tests/          – unit tests
- train.py        – entry point for training
- evaluate.py     – evaluation script
- requirements.txt, LICENSE, .gitignore

src/ subdirectories:
- data/           – dataloader, tokenizer
- model/          – Transformer blocks (attention, embeddings, transformer)
- train/          – config loader, Trainer class
- utils/          – debug helpers, model export

##### README Insights
- Describes end-to-end workflow: Prompt → Tokenize → Embed → Decoder×6 → LM head → Softmax → Generation
- Highlights core components: Tokenizer, Embeddings, Decoder Blocks, Generation loop
- Lists tech stack: Python, PyTorch, Gradio for deployment
- Provides model specs and evaluation metrics

a. Main Components
1. Data Layer
- raw corpus (data/tinyshakespeare.txt)
- configs (configs/*.yaml)
2. Preprocessing
- Tokenizer (src/data/tokenizer.py)
- Dataloader (src/data/dataloader.py)
3. Model
- Input Embeddings + Positional Embeddings (src/model/embeddings.py)
- Decoder Block ×6 (src/model/attention.py & transformer.py)
- LM Head (linear projection in transformer.py)
4. Training Pipeline
- Config Loader (src/train/config.py)
- Trainer (src/train/trainer.py)
- Training loop orchestration (train.py)
- Checkpointing & Metrics output (evaluation_metrics.json)
5. Evaluation
- evaluate.py
- test suite (tests/*.py)
6. Deployment
- Gradio App (deploy/app.py)
- Export logic (src/utils/export.py)

b. Relationships & Data Flow
1. Raw text → Tokenizer → Token IDs
2. Token IDs + Positions → Embedding layer → Tensor [seq_len×embed_dim]
3. Embedded tensor → Decoder stack → Final hidden states
4. Hidden states → LM head → Probability distribution → Next-token sampling
5. Trainer orchestrates 1–4 in batches, updates model via PyTorch optimizers
6. After training, model + tokenizer exported → Gradio app loads them → serves inference requests

c. Architectural Patterns & Principles
- Modular layering: clear separation of data ingestion, modeling, training, and deployment
- Encoder-only style (decoder-only Transformer)
- Configuration-driven experiments (YAML under configs/)
- Test-driven: unit tests for each core component
- Reusable utilities: debug and export helpers

d. Key Technologies
- PyTorch (model, training loop)
- YAML (experiment/config management)
- Gradio (deployment UI)
- Python packaging & scripts (train.py, evaluate.py)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data & Config Layer
    subgraph "Data & Config Layer"
        RawCorp["Raw Corpus (tinyshakespeare.txt)"]:::data
        Configs["Config Files (debug.yaml, gpt1_char.yaml)"]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing"
        Tokenizer["Tokenizer"]:::processing
        Dataloader["Dataloader"]:::processing
    end

    %% Model Module
    subgraph "Model Module"
        Embeddings["Embeddings (Input + Positional)"]:::model
        DecoderBlock["Decoder Block (Attention & MLP)"]:::model
        TransformerStack["Transformer Stack & LM Head"]:::model
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        TrainScript["Training Script (train.py)"]:::processing
        ConfigLoader["Config Loader"]:::processing
        Trainer["Trainer Class"]:::processing
        Checkpoint["Checkpoint & Metrics Output"]:::data
    end

    %% Evaluation
    subgraph "Evaluation"
        EvaluateScript["Evaluation Script"]:::processing
        subgraph "Unit Tests"
            TestTok["test_tokenizer.py"]:::test
            TestAtt["test_attention.py"]:::test
            TestGen["test_generation.py"]:::test
        end
    end

    %% Deployment
    subgraph "Deployment"
        ExportUtil["Model Export Utility"]:::util
        GradioApp["Gradio App"]:::framework
    end

    %% External Frameworks
    PyTorch["PyTorch"]:::framework

    %% Data Flow
    RawCorp -->|"text"| Tokenizer
    Configs -->|"hyperparams"| ConfigLoader
    Configs -->|"hyperparams"| Dataloader
    Tokenizer -->|"[char_ids]"| Dataloader
    Dataloader -->|"[char_ids]"| Embeddings
    Embeddings -->|"[seq_len×embed_dim]"| DecoderBlock
    DecoderBlock -->|"hidden states"| TransformerStack

    %% Training Flow
    TrainScript --> ConfigLoader
    TrainScript --> Trainer
    ConfigLoader --> Trainer
    Trainer --> TransformerStack
    TransformerStack --> Trainer
    Trainer -->|“logs metrics”| Checkpoint
    Trainer -->|“optimizes”| PyTorch
    TransformerStack -->|“constructed in”| PyTorch

    %% Evaluation Flow
    EvaluateScript --> TransformerStack
    EvaluateScript -->|“loads model”| Checkpoint
    TestTok --> Tokenizer
    TestAtt --> DecoderBlock
    TestGen --> TransformerStack

    %% Deployment Flow
    TransformerStack --> ExportUtil
    ExportUtil --> GradioApp
    GradioApp -->|"inference request"| TransformerStack

    %% Styles
    classDef data fill:#bbdefb,stroke:#1e88e5
    classDef processing fill:#c8e6c9,stroke:#43a047
    classDef model fill:#ffcc80,stroke:#fb8c00
    classDef framework fill:#d1c4e9,stroke:#8e24aa
    classDef util fill:#f0f4c3,stroke:#c0ca33
    classDef test fill:#ffcdd2,stroke:#e53935

    %% Click Events
    click RawCorp "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/data/tinyshakespeare.txt"
    click Configs "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/configs/debug.yaml"
    click Tokenizer "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/data/tokenizer.py"
    click Dataloader "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/data/dataloader.py"
    click Embeddings "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/model/embeddings.py"
    click DecoderBlock "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/model/attention.py"
    click TransformerStack "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/model/transformer.py"
    click ConfigLoader "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/train/config.py"
    click Trainer "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/train/trainer.py"
    click TrainScript "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/train.py"
    click EvaluateScript "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/evaluate.py"
    click TestTok "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/tests/test_tokenizer.py"
    click TestAtt "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/tests/test_attention.py"
    click TestGen "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/tests/test_generation.py"
    click ExportUtil "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/src/utils/export.py"
    click GradioApp "https://github.com/ayushsyntax/gpt-from-scratch-in-pytorch/blob/main/deploy/app.py"
```
