How liam-ilan/gpt-rs works
Project Type and Purpose
• This is a command-line, full-stack “ML toolbox” written in Rust. It implements a causal-decoder Transformer (GPT-style) plus a BPE tokenizer, built on top of the tch-rs Rust bindings for libtorch.
• It provides three primary commands:
• tokenize to convert raw text (Parquet) → token IDs (.ron)
• train to consume token IDs + hyperparameter configs → model weights (.safetensors)
• generate to consume model weights + tokenizer + input prompt → generate text
Top-Level File & Directory Analysis
• .cargo/, Cargo.toml/Cargo.lock: Rust package and build configuration
• justfile: CLI script definitions for environment setup
• example/: sample configs and artifactual data (dataset, transformer/train .ron & .safetensors)
• stories.parquet: raw toy dataset
• src/: core Rust code, broken into:
• tokenizer.rs: BPE tokenizer and dataset serialization
• model.rs: Transformer block definitions and forward-pass logic
• train.rs: training loop, data loading, optimizer setup
• generate.rs: inference loop, sampling code
• main.rs: CLI argument parsing, dispatch to subcommands
Key Architectural Patterns & Design Principles
• Pipeline pattern: raw data → tokenization → training → inference
• Modular separation of concerns: Tokenizer, Model, CLI, Config
• Functional style for model forward/backward; State-carrying structs for transformer config
• Plain-text configuration via RON for reproducibility/extensibility
Main Components & Relationships
A. CLI Layer (main.rs)
– Parses arguments & flags (dataset file, config files, seed, device)
– Routes to subcommands: Tokenize, Train, Generate
B. Tokenizer Module (tokenizer.rs)
– Reads Parquet → applies BPE merges → prefixes spaces → outputs .ron token IDs
– Stores vocabulary and merge rules in memory from transformer_config.ron
C. Training Module (train.rs)
– Loads tokenized dataset from .ron file
– Instantiates Transformer model per transformer_config.ron
– Sets up AdamW optimizer, optional MPS device
– Iterates batches → forward/backward → checkpoint to .safetensors
D. Model Core (model.rs)
– Defines Transformer, Block, MultiHeadAttention, FeedForward, Pre-LN + dropout
– Exposes forward pass API for both train & generate
E. Generation Module (generate.rs)
– Loads final .safetensors weights + same model config
– Accepts user prompt → encodes via Tokenizer → autoregressive sampling → decode tokens → text
F. Data Stores & Configs
– Raw: stories.parquet (Parquet dataset)
– Tokenized: datasets/*.ron (Rusty-Object-Notation)
– Model weights: .safetensors
– Hyperparameters: example/train_config.ron, example/transformer_config.ron