# liam-ilan/gpt-rs architecture

> Architecture diagram of the GitHub repository liam-ilan/gpt-rs, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liam-ilan/gpt-rs
- Repository: https://github.com/liam-ilan/gpt-rs

## Overview

##### Project Type and Purpose
- This is a command-line, full-stack “ML toolbox” written in Rust. It implements a causal-decoder Transformer (GPT-style) plus a BPE tokenizer, built on top of the `tch-rs` Rust bindings for libtorch.
- It provides three primary commands:
- `tokenize` to convert raw text (Parquet) → token IDs (.ron)
- `train` to consume token IDs + hyperparameter configs → model weights (.safetensors)
- `generate` to consume model weights + tokenizer + input prompt → generate text

##### Top-Level File & Directory Analysis
- `.cargo/`, `Cargo.toml`/`Cargo.lock`: Rust package and build configuration
- `justfile`: CLI script definitions for environment setup
- `example/`: sample configs and artifactual data (dataset, transformer/train .ron & .safetensors)
- `stories.parquet`: raw toy dataset
- `src/`: core Rust code, broken into:
- `tokenizer.rs`: BPE tokenizer and dataset serialization
- `model.rs`: Transformer block definitions and forward-pass logic
- `train.rs`: training loop, data loading, optimizer setup
- `generate.rs`: inference loop, sampling code
- `main.rs`: CLI argument parsing, dispatch to subcommands

##### Key Architectural Patterns & Design Principles
- Pipeline pattern: raw data → tokenization → training → inference
- Modular separation of concerns: Tokenizer, Model, CLI, Config
- Functional style for model forward/backward; State-carrying structs for transformer config
- Plain-text configuration via RON for reproducibility/extensibility

##### Main Components & Relationships
A. CLI Layer (`main.rs`)
– Parses arguments & flags (dataset file, config files, seed, device)
– Routes to subcommands: Tokenize, Train, Generate
B. Tokenizer Module (`tokenizer.rs`)
– Reads Parquet → applies BPE merges → prefixes spaces → outputs `.ron` token IDs
– Stores vocabulary and merge rules in memory from `transformer_config.ron`
C. Training Module (`train.rs`)
– Loads tokenized dataset from `.ron` file
– Instantiates Transformer model per `transformer_config.ron`
– Sets up AdamW optimizer, optional MPS device
– Iterates batches → forward/backward → checkpoint to `.safetensors`
D. Model Core (`model.rs`)
– Defines `Transformer`, `Block`, `MultiHeadAttention`, `FeedForward`, Pre-LN + dropout
– Exposes forward pass API for both train & generate
E. Generation Module (`generate.rs`)
– Loads final `.safetensors` weights + same model config
– Accepts user prompt → encodes via Tokenizer → autoregressive sampling → decode tokens → text
F. Data Stores & Configs
– Raw: `stories.parquet` (Parquet dataset)
– Tokenized: `datasets/*.ron` (Rusty-Object-Notation)
– Model weights: `.safetensors`
– Hyperparameters: `example/train_config.ron`, `example/transformer_config.ron`

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Modules
    CLI["CLI Dispatcher\n(src/main.rs)"]:::codeModule
    Tokenizer["Tokenizer Module\n(src/tokenizer.rs)"]:::codeModule
    Train["Training Engine\n(src/train.rs)"]:::codeModule
    Generate["Inference Engine\n(src/generate.rs)"]:::codeModule
    Model["Transformer Core\n(src/model.rs)"]:::codeModule

    %% Data Stores
    RawData["Raw Dataset\n(stories.parquet)"]:::dataStore
    TokenData["Tokenized Dataset\n(example/*.ron)"]:::dataStore
    Weights["Model Weights\n(.safetensors)"]:::dataStore

    %% Configs & Scripts
    TrainConfig["Train Config\n(.ron)"]:::config
    TransfConfig["Transformer Config\n(.ron)"]:::config
    Cargo["Cargo Config\n(Cargo.toml/Cargo.lock)"]:::config
    Just["Justfile"]:::config

    %% External
    LibTorch["libtorch / tch-rs"]:::external

    %% Flow
    CLI -->|"tokenize cmd"| Tokenizer
    Tokenizer -->|"writes .ron"| TokenData
    CLI -->|"train cmd"| Train
    Train -->|"loads .ron"| TokenData
    Train -->|"uses Train Config"| TrainConfig
    Train -->|"uses Transformer Config"| TransfConfig
    Train -->|"forwards/backwards"| Model
    Model -->|"writes .safetensors"| Weights
    CLI -->|"generate cmd"| Generate
    Generate -->|"loads weights"| Weights
    Generate -->|"uses Transformer Config"| TransfConfig
    Generate -->|"forwards"| Model
    Generate -->|"outputs text"| CLI
    Tokenizer -->|"uses Transformer Config"| TransfConfig

    %% External dependencies
    CLI --> LibTorch
    Tokenizer --> LibTorch
    Train --> LibTorch
    Generate --> LibTorch
    Model --> LibTorch

    %% Bootstrap
    Cargo --> CLI
    Just --> CLI

    %% Legend
    subgraph Legend
        A["Blue: Code Module"]:::codeModule
        B["Green: Data Store"]:::dataStore
        C["Purple: Config/Script"]:::config
        D["Orange: External Lib"]:::external
    end

    %% Click events
    click CLI "https://github.com/liam-ilan/gpt-rs/blob/main/src/main.rs"
    click Tokenizer "https://github.com/liam-ilan/gpt-rs/blob/main/src/tokenizer.rs"
    click Train "https://github.com/liam-ilan/gpt-rs/blob/main/src/train.rs"
    click Generate "https://github.com/liam-ilan/gpt-rs/blob/main/src/generate.rs"
    click Model "https://github.com/liam-ilan/gpt-rs/blob/main/src/model.rs"
    click RawData "https://github.com/liam-ilan/gpt-rs/blob/main/stories.parquet"
    click TokenData "https://github.com/liam-ilan/gpt-rs/blob/main/example/dataset_2048_stories.parquet.ron"
    click Weights "https://github.com/liam-ilan/gpt-rs/blob/main/example/trained_transformer.safetensors"
    click TrainConfig "https://github.com/liam-ilan/gpt-rs/blob/main/example/train_config.ron"
    click TransfConfig "https://github.com/liam-ilan/gpt-rs/blob/main/example/transformer_config.ron"
    click Cargo "https://github.com/liam-ilan/gpt-rs/blob/main/Cargo.toml"
    click Just "https://github.com/liam-ilan/gpt-rs/tree/main/justfile"

    %% Styles
    classDef codeModule fill:#add8e6,stroke:#000,stroke-width:1.5px
    classDef dataStore fill:#98fb98,stroke:#000,stroke-width:1.5px,shape:cylinder
    classDef external fill:#ffa500,stroke:#000,stroke-width:1.5px,shape:cloud
    classDef config fill:#dda0dd,stroke:#000,stroke-width:1.5px,shape:parallelogram
```
