How tekaratzas/rustgpt works
Project type and purpose
• This is a pure-Rust, from-scratch implementation of a transformer-based large language model (LLM).
• It’s both a library (src/lib.rs + modules) and a CLI/training application (src/main.rs) providing pre-training, instruction tuning, and interactive chat.
File structure
• Top-level:
• Cargo.toml / Cargo.lock → Rust project manifest and dependencies (ndarray, rand)
• README.md → project overview, architecture, training details
• src/ → core code modules
• tests/ → unit tests for each component
• src modules follow a clear modular breakdown:
• vocab.rs (tokenization & vocabulary)
• embeddings.rs (token embedding layer)
• transformer.rs → orchestrates one Transformer block
• self_attention.rs, feed_forward.rs, layer_norm.rs → sub-layers of the Transformer block
• output_projection.rs (final linear layer → vocabulary logits)
• adam.rs (Adam optimizer + gradient clipping)
• llm.rs (core LLM: forward/backward, training loop abstractions)
• main.rs (application entrypoint: data prep, two training phases, interactive REPL)
README insights
• High-level data flow:
Input Text → Tokenization → Embeddings → Transformer Blocks (×3) → Output Projection → Predictions
• Training pipeline phases:
• Pre-training on factual text (cross-entropy loss)
• Instruction tuning on conversational data
• Key hyperparameters and loss/optimizer details
Components and interactions
1. Data Sources
• Factual dataset
• Instruction tuning dataset
2. Vocabulary & Tokenization (vocab.rs)
3. Embedding Layer (embeddings.rs)
4. Transformer Block (transformer.rs)
– Self-Attention (self_attention.rs)
– Feed-Forward (feed_forward.rs)
– Layer Norm (layer_norm.rs)
5. Output Projection (output_projection.rs)
6. Core LLM Controller (llm.rs)
7. Optimizer & Training Loop (adam.rs + backprop logic)
8. CLI / Interactive REPL (main.rs)
9. Tests (tests/…): illustrate coverage but can be grouped as “Unit Tests”
• Data Sources → Vocabulary & Tokenization → token IDs
• token IDs → Embedding Layer → embeddings
• embeddings → [×3 Transformer Block] → hidden states
• hidden states → Output Projection → logits → Predictions
• Predictions + Targets → Loss Function
• Loss → Backprop (through Output Projection, Transformer Blocks, Embeddings)
• Gradients → Adam Optimizer → parameter updates
• LLM Controller orchestrates forward & backward passes per batch
• CLI / Interactive REPL calls LLM Controller in inference mode (greedy decoding)
c. Architectural patterns & principles
• Modular layering: each neural-net component in its own module
• Separation of concerns:
• llm.rs contains high-level training/inference loops
• main.rs is purely I/O and orchestration
• Clean dependency flow: data → model → optimizer → updated model
• Pure Rust + ndarray ensures zero external ML dependencies
d. Technologies & frameworks
• Rust (Cargo)
• ndarray (matrix ops)
• rand / rand_distr (initialization)