How lansslu/llms-from-scratch works
Project Type and Purpose
• This is an educational code repository accompanying the book "Build a Large Language Model (From Scratch)".
• It walks readers step by step through building, pretraining, and finetuning a GPT-style LLM using Python and PyTorch.
• It includes interactive Jupyter notebooks, standalone Python scripts, tests, packaging code (in pkg/llms_from_scratch), environment setup guides, and CI/CD workflows.
File Structure
• Top-level directories:
• ch01–ch07: Chapter folders each containing a 01_main-chapter-code subfolder (notebooks, scripts), plus bonus materials in numbered subfolders.
• appendix-A, appendix-D, appendix-E: Appendix code and notebooks.
• pkg/llms_from_scratch: The core Python package implementing model architectures and utilities, with unit tests under pkg/llms_from_scratch/tests.
• setup: Environment-setup recommendations (Python tips, Docker, AWS SageMaker).
• .github/workflows: CI/CD YAML files for testing on Linux, macOS, Windows, pip compatibility, link checking, linting, spelling.
• Configuration files: pyproject.toml, pixi.toml, requirements.txt, CITATION.cff, LICENSE.txt.
• Architectural patterns:
• Modular packaging: core library under pkg, separate from tutorial notebooks.
• Layered/design-by-example: “UI” notebooks and scripts on top of the core package.
• CI/CD pipeline automating tests across environments.
• Decoupled setup tooling in setup, enabling optional Docker or AWS configurations.
README Insights
• The README outlines chapter progression (from data handling to attention mechanisms to GPT model implementation, training, and finetuning).
• Highlights external dependencies: Python, PyTorch, optional Docker, AWS Sagemaker, tokenizers, and potential use of OpenAI or Ollama for evaluation.
• Points to CI badges demonstrating automated test coverage.
a. Main Components
• Repository Root
• Environment Setup Layer (setup)
• Core Library (pkg/llms_from_scratch)
• Chapter Code Layers
• Data Preparation (tokenizers, dataloaders in ch02)
• Model Definition (attention modules, GPT implementation in ch03/ch04)
• Training Loop & Pretraining (ch05 scripts and notebooks)
• Finetuning Modules (ch06, ch07: classification and instruction tuning)
• Appendices (PyTorch intro, LoRA, training bells and whistles)
• Packaging & Distribution (PyPI via pyproject.toml)
• CI/CD Pipeline (.github/workflows)
• External Services & Libraries
• PyTorch, Tiktoken/BPE, Docker, AWS SageMaker, GitHub Actions, OpenAI/Ollama APIs
b. Relationships and Interactions
• Environment Setup → prepares Python/Docker/AWS environments.
• Chapter Notebooks & Scripts → import and call Core Library functions.
• Core Library → implements model architecture, training/inference utilities, tokenizer abstractions.
• Data Preparation → feeds data into training loops in Core Library.
• Training & Finetuning → produce model artifacts (weights) stored locally or loaded from pretrained sources.
• Packaging → wraps Core Library for reuse and testing.
• CI/CD → automatically runs tests on Core Library, example scripts, and notebooks across platforms.
• External APIs & Services → used in model evaluation (OpenAI/Ollama), optional AWS SageMaker execution.
c. Architectural Patterns and Principles
• Separation of concerns: clear division between tutorials (notebooks), core logic (library), and setup/config.
• Layered architecture: from low-level utilities to high-level training and evaluation.
• Reusability: packaging model components for unit testing and reuse.
• Extensibility: optional bonus materials and user interface scripts that plug into the core.
d. Key Technologies and Frameworks
• Python 3.x, PyTorch, Jupyter Notebooks
• BPE Tokenizers (custom and tiktoken)
• GitHub Actions for CI/CD
• Docker (optional dev containers)
• AWS SageMaker (optional notebook environments)