How yh1zzzzzzz/llm-from-scratch works
Main High-Level Components
• Data Acquisition & Preprocessing
• Data_Crawl (HTML→text, deduplication, filtering)
• Training_Data (raw corpora, checkpointed data)
• Tokenization & Embedding
• BPE Pretokenizer/Encoder/Decoder (serial & parallel variants)
• Token Embedding + Rotary Position Embedding
• Core Compute Operators
• Operator/Attention (Naïve, FlashAttention)
• Operator/MatMul (NumPy, PyTorch, Triton, CUDA)
• Operator/Softmax, Normalization, etc.
• activation_function (SwiGLU, GELU, ReLU in Triton/CUDA)
• Model Architecture
• Architecture/model.py (FFN, MultiHeadAttention, RMSNorm)
• nn_utils.py (layer linking, auto-grad wrappers)
• Loss & Optimization
• Loss_function/Cross_entropy.py
• Optimizer (AdamW, SGD, schedulers, grad_clipping)
• Training Pipeline
• Training/train.py, run_training.sh, generate.py
• CheckPoint.py, DataLoader, training loop
• Benchmark & Profiler
• benchmark&profiler for performance measurement
• Advanced/Planned Modules
• Alignment (reward-finetuning, reasoning improvements)
• MoE (Mixture of Experts)
• Parallelsim (distributed data parallel)
• Scaling heuristics
Relationships & Data Flow
• Raw data flows → Data_Crawl → preprocessed text files
• Preprocessed text → Tokenize&Embedding → token IDs & embeddings
• Embeddings → passed into Transformer layers built from Core Operators
• Forward pass outputs → Loss module → backward pass → Optimizer
• Training loop orchestrates forward/backward/checkpoint cycles
• Benchmark & Profiler instrument compute modules for performance data
• Alignment, MoE, Parallelsim plug into the training loop as optional branches
Architectural Patterns & Design Principles
• Layered architecture:
1. Low-level compute kernels (CUDA/Triton/PyTorch)
2. Core operator abstractions (torch.autograd compatible wrappers)
3. Model composition (Transformer blocks)
4. Training orchestration
• Clear separation of concerns: preprocessing, compute, architecture, training, benchmarking
Key Technologies & Libraries
• Python ≥3.8, PyTorch ≥2.0 (for autograd and tensor ops)
• CUDA & Triton (for custom high-performance kernels)
• NumPy (reference implementations)
• Shell scripts (run_training.sh, run_generation.sh)
• Optional: Matplotlib/Seaborn for profiling outputs
Checklist Before You Finish
• Is the layering and modular separation visually obvious?