# yh1zzzzzzz/llm-from-scratch architecture

> Architecture diagram of the GitHub repository yh1zzzzzzz/llm-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yh1zzzzzzz/llm-from-scratch
- Repository: https://github.com/yh1zzzzzzz/llm-from-scratch

## Overview

##### Main High-Level Components
- Data Acquisition & Preprocessing
- Data_Crawl (HTML→text, deduplication, filtering)
- Training_Data (raw corpora, checkpointed data)
- Tokenization & Embedding
- BPE Pretokenizer/Encoder/Decoder (serial & parallel variants)
- Token Embedding + Rotary Position Embedding
- Core Compute Operators
- Operator/Attention (Naïve, FlashAttention)
- Operator/MatMul (NumPy, PyTorch, Triton, CUDA)
- Operator/Softmax, Normalization, etc.
- activation_function (SwiGLU, GELU, ReLU in Triton/CUDA)
- Model Architecture
- Architecture/model.py (FFN, MultiHeadAttention, RMSNorm)
- nn_utils.py (layer linking, auto-grad wrappers)
- Loss & Optimization
- Loss_function/Cross_entropy.py
- Optimizer (AdamW, SGD, schedulers, grad_clipping)
- Training Pipeline
- Training/train.py, run_training.sh, generate.py
- CheckPoint.py, DataLoader, training loop
- Benchmark & Profiler
- benchmark&profiler for performance measurement
- Advanced/Planned Modules
- Alignment (reward-finetuning, reasoning improvements)
- MoE (Mixture of Experts)
- Parallelsim (distributed data parallel)
- Scaling heuristics

##### Relationships & Data Flow
- Raw data flows → Data_Crawl → preprocessed text files
- Preprocessed text → Tokenize&Embedding → token IDs & embeddings
- Embeddings → passed into Transformer layers built from Core Operators
- Forward pass outputs → Loss module → backward pass → Optimizer
- Training loop orchestrates forward/backward/checkpoint cycles
- Benchmark & Profiler instrument compute modules for performance data
- Alignment, MoE, Parallelsim plug into the training loop as optional branches

##### Architectural Patterns & Design Principles
- Layered architecture:
1. Low-level compute kernels (CUDA/Triton/PyTorch)
2. Core operator abstractions (torch.autograd compatible wrappers)
3. Model composition (Transformer blocks)
4. Training orchestration
- Clear separation of concerns: preprocessing, compute, architecture, training, benchmarking

##### Key Technologies & Libraries
- Python ≥3.8, PyTorch ≥2.0 (for autograd and tensor ops)
- CUDA & Triton (for custom high-performance kernels)
- NumPy (reference implementations)
- Shell scripts (run_training.sh, run_generation.sh)
- Optional: Matplotlib/Seaborn for profiling outputs

##### Checklist Before You Finish
- Is the layering and modular separation visually obvious?

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Layer
    subgraph "Data & Preprocessing"
        DataCrawl["Data_Crawl"]:::data
        TrainingData[(Training_Data)]:::data
    end

    %% Tokenization & Embedding Layer
    subgraph "Tokenization & Embedding"
        DataLoader["Data_loader"]:::compute
        SerialPre["Serial_BPE_PreTokenizer"]:::compute
        ParallelPre["Parallel_BPE_PreToken"]:::compute
        Encode["BPE_encode"]:::compute
        Decode["BPE_decode"]:::compute
        CompleteBPE["Complete_BPE"]:::compute
        TokenEmbed["Token_Embedding"]:::compute
        RoPE["RoPE_embedding"]:::compute
    end

    %% Operator Kernels Layer
    subgraph "Operator Kernels"
        NaiveAtt["Naïve Attention"]:::compute
        FlashAtt["FlashAttention"]:::compute
        MMNumpy["MatMul (NumPy)"]:::compute
        MMPy["MatMul (PyTorch)"]:::compute
        MMTriton["MatMul (Triton)"]:::compute
        AddTriton["MatrixAdd (Triton)"]:::compute
        SoftNaive["Softmax (Naïve)"]:::compute
        SoftVec["Softmax (Vectorized)"]:::compute
        SoftTriton["Softmax (Triton)"]:::compute
        LayerNormT["LayerNorm (Triton)"]:::compute
        Activations["activation_function"]:::compute
    end

    %% Model Architecture Layer
    subgraph "Model Architecture"
        Model["model.py"]:::model
        FFN["FFN.py"]:::model
        MHAtt["MultiHeadAtten.py"]:::model
        RMSNorm["RMSNorm.py"]:::model
        NNUtils["nn_utils.py"]:::model
    end

    %% Loss & Optimization Layer
    subgraph "Loss & Optimization"
        Loss["Cross_entropy.py"]:::training
        AdamW["AdamW.py"]:::training
        SGD["SGD.py"]:::training
        Scheduler["cosine_annealing.py"]:::training
        GradClip["grad_clipping.py"]:::training
    end

    %% Training Control Layer
    subgraph "Training Control"
        TrainLoop["train.py"]:::training
        Checkpoint["CheckPoint.py"]:::training
        RunTrain["run_training.sh"]:::training
        RunGen["run_generation.sh"]:::training
        Generate["generate.py"]:::training
    end

    %% Benchmark & Profiler
    Benchmark["benchmark&profiler"]:::external

    %% Planned Modules
    subgraph "Planned Modules" 
        direction TB
        Alignment["Alignment"]:::planned
        MoE["MoE"]:::planned
        Parallelsim["Parallelsim"]:::planned
        Scaling["Scaling Heuristics"]:::planned
    end

    %% Data & Control Flow
    DataCrawl --> TrainingData
    TrainingData --> DataLoader
    DataLoader --> SerialPre
    DataLoader --> ParallelPre
    SerialPre --> Encode
    ParallelPre --> Encode
    Encode --> Decode
    Decode --> CompleteBPE
    CompleteBPE --> TokenEmbed
    TokenEmbed --> RoPE
    RoPE --> Model
    %% Kernels to Model
    NaiveAtt --> Model
    FlashAtt --> Model
    MMNumpy --> Model
    MMPy --> Model
    MMTriton --> Model
    AddTriton --> Model
    SoftNaive --> Model
    SoftVec --> Model
    SoftTriton --> Model
    LayerNormT --> Model
    Activations --> Model
    %% Model to Loss and Optimization
    Model --> Loss
    Loss --> AdamW
    Loss --> SGD
    AdamW --> Scheduler
    SGD --> GradClip
    %% Training orchestration
    Loss --> TrainLoop
    Scheduler --> TrainLoop
    GradClip --> TrainLoop
    TrainLoop --> Checkpoint
    RunTrain --> TrainLoop
    TrainLoop --> Generate
    RunGen --> Generate
    %% Benchmark hooks
    Benchmark -.-> NaiveAtt
    Benchmark -.-> FlashAtt
    Benchmark -.-> TrainLoop
    %% Planned integration
    Alignment -.-> TrainLoop
    MoE -.-> TrainLoop
    Parallelsim -.-> TrainLoop
    Scaling -.-> TrainLoop

    %% Click Events
    click DataCrawl "https://github.com/yh1zzzzzzz/llm-from-scratch/tree/main/Data_Crawl/"
    click TrainingData "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Training/Training_Data/data.txt"
    click SerialPre "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Tokenize&Embedding/Serial_BPE_PreTokenizer.py"
    click ParallelPre "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Tokenize&Embedding/Parallel_BPE_PreToken.py"
    click Encode "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Tokenize&Embedding/BPE_encode.py"
    click Decode "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Tokenize&Embedding/BPE_decode.py"
    click CompleteBPE "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Tokenize&Embedding/Complete_BPE.py"
    click TokenEmbed "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Tokenize&Embedding/Token_Embedding.py"
    click RoPE "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Tokenize&Embedding/RoPE_embedding(naive).py"
    click NaiveAtt "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/Attention/Compile_NaiveAtten.py"
    click FlashAtt "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/Attention/FlashAttention.py"
    click MMNumpy "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/MatirxMul/Tile_MatrixMul(Numpy).py"
    click MMPy "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/MatirxMul/Tile_MatrixMul(Pytorch).py"
    click MMTriton "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/MatirxMul/Triton_MatMul.py"
    click AddTriton "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/MatirxMul/Triton_MatrixAdd.py"
    click SoftNaive "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/Softmax/Online_SoftMax(Naive).py"
    click SoftVec "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/Softmax/Vector_SoftMax.py"
    click SoftTriton "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/Softmax/Triton_softmax.py"
    click LayerNormT "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Operator/Normalization/Triton_LayerNorm.py"
    click Activations "https://github.com/yh1zzzzzzz/llm-from-scratch/tree/main/activation_function/"
    click Model "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Architecture/model.py"
    click FFN "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Architecture/FFN.py"
    click MHAtt "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Architecture/MultiHeadAtten.py"
    click RMSNorm "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Architecture/RMSNorm.py"
    click NNUtils "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Architecture/nn_utils.py"
    click Loss "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Loss_function/Cross_entropy.py"
    click AdamW "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Optimizer/AdamW.py"
    click SGD "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Optimizer/SGD.py"
    click Scheduler "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Optimizer/cosine_annealing.py"
    click GradClip "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Optimizer/grad_clipping.py"
    click TrainLoop "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Training/train.py"
    click Checkpoint "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Training/CheckPoint.py"
    click RunTrain "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Training/run_training.sh"
    click RunGen "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Training/run_generation.sh"
    click Generate "https://github.com/yh1zzzzzzz/llm-from-scratch/blob/main/Training/generate.py"
    click Benchmark "https://github.com/yh1zzzzzzz/llm-from-scratch/tree/main/benchmark&profiler/"
    click Alignment "https://github.com/yh1zzzzzzz/llm-from-scratch/tree/main/Alignment/"
    click MoE "https://github.com/yh1zzzzzzz/llm-from-scratch/tree/main/MoE/"
    click Parallelsim "https://github.com/yh1zzzzzzz/llm-from-scratch/tree/main/Parallelsim/"
    click Scaling "https://github.com/yh1zzzzzzz/llm-from-scratch/tree/main/Scaling/"
    
    %% Styles
    classDef data fill:#dae8fc,stroke:#6c8ebf
    classDef compute fill:#d5e8d4,stroke:#82b366
    classDef model fill:#ffe6cc,stroke:#d79b00
    classDef training fill:#f8cecc,stroke:#b85450
    classDef external fill:#eeeeee,stroke:#888888
    classDef planned stroke-dasharray: 5 5,fill:#ffffff,stroke:#aaaaaa
    class DataCrawl,TrainingData,DataLoader,SerialPre,ParallelPre,Encode,Decode,CompleteBPE,TokenEmbed,RoPE data
    class NaiveAtt,FlashAtt,MMNumpy,MMPy,MMTriton,AddTriton,SoftNaive,SoftVec,SoftTriton,LayerNormT,Activations compute
    class Model,FFN,MHAtt,RMSNorm,NNUtils model
    class Loss,AdamW,SGD,Scheduler,GradClip,TrainLoop,Checkpoint,RunTrain,RunGen,Generate training
    class Benchmark,DataCrawl external
```
