# lansslu/llms-from-scratch architecture

> Architecture diagram of the GitHub repository lansslu/llms-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lansslu/llms-from-scratch
- Repository: https://github.com/lansslu/llms-from-scratch

## Overview

##### Project Type and Purpose
- This is an educational code repository accompanying the book "Build a Large Language Model (From Scratch)".
- It walks readers step by step through building, pretraining, and finetuning a GPT-style LLM using Python and PyTorch.
- It includes interactive Jupyter notebooks, standalone Python scripts, tests, packaging code (in `pkg/llms_from_scratch`), environment setup guides, and CI/CD workflows.

##### File Structure
- Top-level directories:
- `ch01`–`ch07`: Chapter folders each containing a `01_main-chapter-code` subfolder (notebooks, scripts), plus bonus materials in numbered subfolders.
- `appendix-A`, `appendix-D`, `appendix-E`: Appendix code and notebooks.
- `pkg/llms_from_scratch`: The core Python package implementing model architectures and utilities, with unit tests under `pkg/llms_from_scratch/tests`.
- `setup`: Environment-setup recommendations (Python tips, Docker, AWS SageMaker).
- `.github/workflows`: CI/CD YAML files for testing on Linux, macOS, Windows, pip compatibility, link checking, linting, spelling.
- Configuration files: `pyproject.toml`, `pixi.toml`, `requirements.txt`, `CITATION.cff`, `LICENSE.txt`.
- Architectural patterns:
- Modular packaging: core library under `pkg`, separate from tutorial notebooks.
- Layered/design-by-example: “UI” notebooks and scripts on top of the core package.
- CI/CD pipeline automating tests across environments.
- Decoupled setup tooling in `setup`, enabling optional Docker or AWS configurations.

##### README Insights
- The README outlines chapter progression (from data handling to attention mechanisms to GPT model implementation, training, and finetuning).
- Highlights external dependencies: Python, PyTorch, optional Docker, AWS Sagemaker, tokenizers, and potential use of OpenAI or Ollama for evaluation.
- Points to CI badges demonstrating automated test coverage.

a. Main Components
- Repository Root
- Environment Setup Layer (`setup`)
- Core Library (`pkg/llms_from_scratch`)
- Chapter Code Layers
- Data Preparation (tokenizers, dataloaders in ch02)
- Model Definition (attention modules, GPT implementation in ch03/ch04)
- Training Loop & Pretraining (ch05 scripts and notebooks)
- Finetuning Modules (ch06, ch07: classification and instruction tuning)
- Appendices (PyTorch intro, LoRA, training bells and whistles)
- Packaging & Distribution (PyPI via `pyproject.toml`)
- CI/CD Pipeline (.github/workflows)
- External Services & Libraries
- PyTorch, Tiktoken/BPE, Docker, AWS SageMaker, GitHub Actions, OpenAI/Ollama APIs
b. Relationships and Interactions
- Environment Setup → prepares Python/Docker/AWS environments.
- Chapter Notebooks & Scripts → import and call Core Library functions.
- Core Library → implements model architecture, training/inference utilities, tokenizer abstractions.
- Data Preparation → feeds data into training loops in Core Library.
- Training & Finetuning → produce model artifacts (weights) stored locally or loaded from pretrained sources.
- Packaging → wraps Core Library for reuse and testing.
- CI/CD → automatically runs tests on Core Library, example scripts, and notebooks across platforms.
- External APIs & Services → used in model evaluation (OpenAI/Ollama), optional AWS SageMaker execution.
c. Architectural Patterns and Principles
- Separation of concerns: clear division between tutorials (notebooks), core logic (library), and setup/config.
- Layered architecture: from low-level utilities to high-level training and evaluation.
- Reusability: packaging model components for unit testing and reuse.
- Extensibility: optional bonus materials and user interface scripts that plug into the core.
d. Key Technologies and Frameworks
- Python 3.x, PyTorch, Jupyter Notebooks
- BPE Tokenizers (custom and tiktoken)
- GitHub Actions for CI/CD
- Docker (optional dev containers)
- AWS SageMaker (optional notebook environments)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Environment Setup Layer
    subgraph "Environment Setup Layer"
        Setup["Environment Setup"]:::optional
    end

    %% Core Library
    subgraph "Core Library"
        Core["Core Library (pkg/llms_from_scratch)"]:::code
        Tests["Core Library Tests"]:::code
    end

    %% Chapter Code Layers
    subgraph "Chapter Code Layers"
        DataPrep["Data Preparation"]:::code
        BPEBonus["Byte-Pair Encoding Bonus"]:::code
        subgraph "Model Definition"
            MD3["Chapter 3 Code"]:::code
            MD4["Chapter 4 Code"]:::code
        end
        Training["Pretraining & Training Loop"]:::code
        PretrainBonus["Pretraining Bonus"]:::optional
        FinetuneClass["Finetuning - Classification"]:::code
        FinetuneInstr["Finetuning - Instruction Tuning"]:::code
        subgraph "Appendix A"
            AppendixA_Code["Appendix A Code"]:::code
            AppendixA_Setup["Appendix A Setup"]:::optional
        end
        AppendixD["Appendix D"]:::code
        AppendixE["Appendix E"]:::code
    end

    %% Packaging & CI/CD
    Packaging["Packaging & Distribution"]:::code
    CI["CI/CD Pipeline"]:::external

    %% External Services & Libraries
    subgraph "External Services & Libraries"
        Torch["PyTorch"]:::external
        Tokenizers["Tokenizers"]:::external
        Docker["Docker"]:::optional
        Sage["AWS SageMaker"]:::optional
        OpenAI["OpenAI/Ollama APIs"]:::external
    end

    %% Data Artifacts
    Weights["Model Artifacts (Weights)"]:::data

    %% Relationships
    Setup -->|"provisions"| Core
    Setup -->|"provisions"| DataPrep
    Setup -->|"provisions"| MD3
    Setup -->|"provisions"| MD4
    Setup -->|"provisions"| Training
    Setup -->|"provisions"| FinetuneClass
    Setup -->|"provisions"| FinetuneInstr
    Setup -->|"optional"| Docker
    Setup -->|"optional"| Sage

    DataPrep -->|"imports"| Core
    BPEBonus -->|"imports"| Core
    MD3 -->|"imports"| Core
    MD4 -->|"imports"| Core
    Training -->|"imports & trains"| Core
    FinetuneClass -->|"imports & finetunes"| Core
    FinetuneInstr -->|"imports & finetunes"| Core
    AppendixA_Code -->|"demos & imports"| Core
    AppendixA_Setup -->|"optional setups"| Core
    AppendixD -->|"demos & imports"| Core
    AppendixE -->|"demos & imports"| Core

    Core -->|"trains"| Weights
    Training -->|"produces"| Weights
    PretrainBonus -->|"alternative loads"| Weights
    FinetuneClass -->|"produces"| Weights
    FinetuneInstr -->|"produces"| Weights

    Weights -->|"evaluates"| OpenAI

    CI -->|"runs tests"| Core
    CI -->|"runs tests"| DataPrep
    CI -->|"runs tests"| MD3
    CI -->|"runs tests"| MD4
    CI -->|"runs tests"| Training
    CI -->|"runs tests"| Packaging

    %% Click Events
    click Setup "https://github.com/lansslu/llms-from-scratch/tree/main/setup/"
    click Core "https://github.com/lansslu/llms-from-scratch/tree/main/pkg/llms_from_scratch/"
    click Tests "https://github.com/lansslu/llms-from-scratch/tree/main/pkg/llms_from_scratch/tests/"
    click DataPrep "https://github.com/lansslu/llms-from-scratch/tree/main/ch02/01_main-chapter-code/"
    click BPEBonus "https://github.com/lansslu/llms-from-scratch/tree/main/ch02/02_bonus_bytepair-encoder/"
    click MD3 "https://github.com/lansslu/llms-from-scratch/tree/main/ch03/01_main-chapter-code/"
    click MD4 "https://github.com/lansslu/llms-from-scratch/tree/main/ch04/01_main-chapter-code/"
    click Training "https://github.com/lansslu/llms-from-scratch/tree/main/ch05/01_main-chapter-code/"
    click PretrainBonus "https://github.com/lansslu/llms-from-scratch/tree/main/ch05/02_alternative_weight_loading/"
    click FinetuneClass "https://github.com/lansslu/llms-from-scratch/tree/main/ch06/01_main-chapter-code/"
    click FinetuneInstr "https://github.com/lansslu/llms-from-scratch/tree/main/ch07/01_main-chapter-code/"
    click AppendixA_Code "https://github.com/lansslu/llms-from-scratch/tree/main/appendix-A/01_main-chapter-code/"
    click AppendixA_Setup "https://github.com/lansslu/llms-from-scratch/tree/main/appendix-A/02_setup-recommendations/"
    click AppendixD "https://github.com/lansslu/llms-from-scratch/tree/main/appendix-D/01_main-chapter-code/"
    click AppendixE "https://github.com/lansslu/llms-from-scratch/tree/main/appendix-E/01_main-chapter-code/"
    click Packaging "https://github.com/lansslu/llms-from-scratch/blob/main/pyproject.toml"
    click CI "https://github.com/lansslu/llms-from-scratch/tree/main/.github/workflows/"

    %% Styles
    classDef code fill:#E3F2FD,stroke:#1E88E5,color:#0D47A1
    classDef data fill:#E8F5E9,stroke:#43A047,color:#1B5E20
    classDef external fill:#FFF3E0,stroke:#FB8C00,color:#E65100
    classDef optional fill:#EEEEEE,stroke:#757575,color:#424242
```
