# shiva-manu/llms-from-scratch architecture

> Architecture diagram of the GitHub repository shiva-manu/llms-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shiva-manu/llms-from-scratch
- Repository: https://github.com/shiva-manu/llms-from-scratch

## Overview

##### Main System Components

a) Core Model Architecture Layer:
- GPT Model Implementation (from ch04/01_main-chapter-code/gpt.py)
- Attention Mechanisms (from ch03)
- Embedding Components

b) Data Processing Layer:
- Text Tokenization
- BytePair Encoding (from ch02/02_bonus_bytepair-encoder)
- DataLoader Components
- Data flow from raw text to tokenized input

c) Training Infrastructure:
- Pretraining Pipeline
- Finetuning Components (Classification & Instruction)
- Learning Rate Schedulers
- Optimization Components

d) Inference & Deployment:
- Generation Pipeline
- User Interface Components (from ch05/06_user_interface)
- Model Weight Loading Systems
- Both training and inference paths

##### Component Relationships

- Component dependencies (thin lines)

##### Implementation Details

Technical annotations for:
- PyTorch integration points
- GPU acceleration paths
- Key configuration parameters
- Memory optimization points

##### Additional Elements

Include:
- Version compatibility notes
- Hardware requirement indicators
- Optional component markers

##### Optimization Features

Highlight:
- Performance optimization points
- Memory efficiency mechanisms
- Distributed training capabilities
- FLOPS analysis points (from ch04/02_performance-analysis)

This approach will create a comprehensive yet understandable visualization of the project's architecture, suitable for both educational purposes and technical implementation understanding.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style Definitions
    classDef core fill:#b3d1ff,stroke:#333,stroke-width:2px
    classDef data fill:#90EE90,stroke:#333,stroke-width:2px
    classDef training fill:#FFB366,stroke:#333,stroke-width:2px
    classDef inference fill:#DDA0DD,stroke:#333,stroke-width:2px
    classDef optional stroke-dasharray: 5 5

    %% Core Model Architecture Layer
    subgraph CoreArch["Core Model Architecture"]
        GPT["GPT Model"]
        Attention["Multi-head Attention"]
        Embedding["Embedding Layer"]
    end

    %% Data Processing Layer
    subgraph DataProc["Data Processing"]
        Tokenizer["Text Tokenizer"]
        BPE["BytePair Encoder"]
        DataLoader["Data Loader"]
    end

    %% Training Infrastructure
    subgraph Training["Training Infrastructure"]
        Pretrain["Pretraining Pipeline"]
        FineTune["Model Finetuning"]
        LRScheduler["Learning Rate Schedulers"]
        DDP["Distributed Training"]
        InstFT["Instruction Finetuning"]
        DPO["DPO Training"]:::optional
    end

    %% Inference & Deployment
    subgraph Inference["Inference & Deployment"]
        GenPipe["Generation Pipeline"]
        UI["User Interface"]
        WeightLoad["Memory Efficient Loading"]
        Eval["Model Evaluation"]
    end

    %% Performance & Analysis
    subgraph Perf["Performance & Analysis"]
        PerfAnalysis["FLOPS Analysis"]
    end

    %% Relationships
    Tokenizer --> BPE
    BPE --> DataLoader
    DataLoader --> GPT
    Embedding --> GPT
    Attention --> GPT
    GPT --> Pretrain
    GPT --> FineTune
    GPT --> InstFT
    GPT --> DPO
    LRScheduler --> Pretrain
    LRScheduler --> FineTune
    DDP --> Training
    GPT --> GenPipe
    GenPipe --> UI
    WeightLoad --> GenPipe
    FineTune --> Eval
    InstFT --> Eval
    DPO --> Eval

    %% Click Events
    click GPT "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch04/01_main-chapter-code/gpt.py"
    click Attention "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch03/01_main-chapter-code/multihead-attention.ipynb"
    click BPE "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py"
    click Pretrain "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py"
    click LRScheduler "https://github.com/shiva-manu/LLMs-from-scratch/tree/main/ch05/04_learning_rate_schedulers"
    click FineTune "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch06/01_main-chapter-code/gpt_class_finetune.py"
    click DPO "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb"
    click GenPipe "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_generate.py"
    click UI "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch05/06_user_interface/app_own.py"
    click WeightLoad "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch05/08_memory_efficient_weight_loading/memory-efficient-state-dict.ipynb"
    click PerfAnalysis "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch04/02_performance-analysis/flops-analysis.ipynb"
    click DDP "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/appendix-A/01_main-chapter-code/DDP-script.py"
    click InstFT "https://github.com/shiva-manu/LLMs-from-scratch/blob/main/ch07/01_main-chapter-code/gpt_instruction_finetuning.py"
    click Eval "https://github.com/shiva-manu/LLMs-from-scratch/tree/main/ch07/03_model-evaluation"

    %% Style Applications
    class GPT,Attention,Embedding core
    class Tokenizer,BPE,DataLoader data
    class Pretrain,FineTune,LRScheduler,DDP,InstFT,DPO training
    class GenPipe,UI,WeightLoad,Eval inference
    class DPO optional
```
