# tekaratzas/rustgpt architecture

> Architecture diagram of the GitHub repository tekaratzas/rustgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tekaratzas/rustgpt
- Repository: https://github.com/tekaratzas/rustgpt

## Overview

##### Project type and purpose
- This is a pure-Rust, from-scratch implementation of a transformer-based large language model (LLM).
- It’s both a library (`src/lib.rs` + modules) and a CLI/training application (`src/main.rs`) providing pre-training, instruction tuning, and interactive chat.

##### File structure
- Top-level:
- Cargo.toml / Cargo.lock → Rust project manifest and dependencies (`ndarray`, `rand`)
- README.md → project overview, architecture, training details
- src/ → core code modules
- tests/ → unit tests for each component
- src modules follow a clear modular breakdown:
- vocab.rs (tokenization & vocabulary)
- embeddings.rs (token embedding layer)
- transformer.rs → orchestrates one Transformer block
- self_attention.rs, feed_forward.rs, layer_norm.rs → sub-layers of the Transformer block
- output_projection.rs (final linear layer → vocabulary logits)
- adam.rs (Adam optimizer + gradient clipping)
- llm.rs (core LLM: forward/backward, training loop abstractions)
- main.rs (application entrypoint: data prep, two training phases, interactive REPL)

##### README insights
- High-level data flow:
Input Text → Tokenization → Embeddings → Transformer Blocks (×3) → Output Projection → Predictions
- Training pipeline phases:
- Pre-training on factual text (cross-entropy loss)
- Instruction tuning on conversational data
- Key hyperparameters and loss/optimizer details

##### Components and interactions
1. Data Sources
- Factual dataset
- Instruction tuning dataset
2. Vocabulary & Tokenization (vocab.rs)
3. Embedding Layer (embeddings.rs)
4. Transformer Block (transformer.rs)
– Self-Attention (self_attention.rs)
– Feed-Forward (feed_forward.rs)
– Layer Norm (layer_norm.rs)
5. Output Projection (output_projection.rs)
6. Core LLM Controller (llm.rs)
7. Optimizer & Training Loop (adam.rs + backprop logic)
8. CLI / Interactive REPL (main.rs)
9. Tests (tests/…): illustrate coverage but can be grouped as “Unit Tests”

- Data Sources → Vocabulary & Tokenization → token IDs
- token IDs → Embedding Layer → embeddings
- embeddings → [×3 Transformer Block] → hidden states
- hidden states → Output Projection → logits → Predictions
- Predictions + Targets → Loss Function
- Loss → Backprop (through Output Projection, Transformer Blocks, Embeddings)
- Gradients → Adam Optimizer → parameter updates
- LLM Controller orchestrates forward & backward passes per batch
- CLI / Interactive REPL calls LLM Controller in inference mode (greedy decoding)

c. Architectural patterns & principles
- Modular layering: each neural-net component in its own module
- Separation of concerns:
- `llm.rs` contains high-level training/inference loops
- `main.rs` is purely I/O and orchestration
- Clean dependency flow: data → model → optimizer → updated model
- Pure Rust + `ndarray` ensures zero external ML dependencies

d. Technologies & frameworks
- Rust (Cargo)
- ndarray (matrix ops)
- rand / rand_distr (initialization)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data I/O Layer
    DataSources["Data Sources\n• Factual dataset\n• Instruction dataset"]:::data
    DataSources -->|“data batches”| Tokenization

    %% Model Layer
    Tokenization["Vocabulary & Tokenization"]:::model
    click Tokenization "https://github.com/tekaratzas/rustgpt/blob/main/src/vocab.rs"
    Tokenization -->|“token IDs”| Embedding

    Embedding["Embedding Layer"]:::model
    click Embedding "https://github.com/tekaratzas/rustgpt/blob/main/src/embeddings.rs"
    Embedding -->|“embeddings”| TransformerBlock

    subgraph "Transformer Block" 
        direction TB
        TB_Core["Transformer Block"]:::model
        click TB_Core "https://github.com/tekaratzas/rustgpt/blob/main/src/transformer.rs"
        TB_Core --> SelfAttention
        TB_Core --> FeedForward
        TB_Core --> LayerNorm

        SelfAttention["Self-Attention"]:::model
        click SelfAttention "https://github.com/tekaratzas/rustgpt/blob/main/src/self_attention.rs"

        FeedForward["Feed-Forward"]:::model
        click FeedForward "https://github.com/tekaratzas/rustgpt/blob/main/src/feed_forward.rs"

        LayerNorm["Layer Norm"]:::model
        click LayerNorm "https://github.com/tekaratzas/rustgpt/blob/main/src/layer_norm.rs"
    end
    TransformerBlock -->|“hidden states”| OutputProjection

    OutputProjection["Output Projection"]:::model
    click OutputProjection "https://github.com/tekaratzas/rustgpt/blob/main/src/output_projection.rs"
    OutputProjection -->|“logits”| Predictions

    Predictions["Predictions"]:::model
    Predictions -->|“predictions + targets”| Loss

    %% Training Logic Layer
    Loss["Loss Computation"]:::train
    Loss -->|"gradients"| CoreLLM

    CoreLLM["Core LLM Controller"]:::train
    click CoreLLM "https://github.com/tekaratzas/rustgpt/blob/main/src/llm.rs"
    CoreLLM -->|"forward/backward"| Tokenization
    CoreLLM -->|"updates"| Optimizer

    Optimizer["Optimizer & Training Loop"]:::train
    click Optimizer "https://github.com/tekaratzas/rustgpt/blob/main/src/adam.rs"
    Optimizer -->|"updated params"| CoreLLM

    %% Hyperparameters annotation
    subgraph "Hyperparameters"
        H["vocab size, embed dim, block count"]:::note
    end
    H -.-> TransformerBlock

    %% CLI / Interactive
    CLI["CLI / Interactive REPL"]:::cli
    click CLI "https://github.com/tekaratzas/rustgpt/blob/main/src/main.rs"
    CLI -->|"inference calls"| CoreLLM

    %% Library Root
    LibraryRoot["Library Root"]:::model
    click LibraryRoot "https://github.com/tekaratzas/rustgpt/blob/main/src/lib.rs"

    %% Unit Tests
    subgraph "Unit Tests" 
        direction TB
        TVocab["vocab_test.rs"]:::test
        click TVocab "https://github.com/tekaratzas/rustgpt/blob/main/tests/vocab_test.rs"
        TEmbed["embeddings_test.rs"]:::test
        click TEmbed "https://github.com/tekaratzas/rustgpt/blob/main/tests/embeddings_test.rs"
        TSA["self_attention_test.rs"]:::test
        click TSA "https://github.com/tekaratzas/rustgpt/blob/main/tests/self_attention_test.rs"
        TFF["feed_forward_test.rs"]:::test
        click TFF "https://github.com/tekaratzas/rustgpt/blob/main/tests/feed_forward_test.rs"
        TLN["layer_norm_test.rs"]:::test
        click TLN "https://github.com/tekaratzas/rustgpt/blob/main/tests/layer_norm_test.rs"
        TTrans["transformer_test.rs"]:::test
        click TTrans "https://github.com/tekaratzas/rustgpt/blob/main/tests/transformer_test.rs"
        TOut["output_projection_test.rs"]:::test
        click TOut "https://github.com/tekaratzas/rustgpt/blob/main/tests/output_projection_test.rs"
        TLLM["llm_test.rs"]:::test
        click TLLM "https://github.com/tekaratzas/rustgpt/blob/main/tests/llm_test.rs"
        TOpt["adam_test.rs"]:::test
        click TOpt "https://github.com/tekaratzas/rustgpt/blob/main/tests/adam_test.rs"
    end

    %% Styles
    classDef data fill:#cce5ff,stroke:#333,stroke-width:1px;
    classDef model fill:#d4edda,stroke:#333,stroke-width:1px;
    classDef train fill:#fff3cd,stroke:#333,stroke-width:1px;
    classDef cli fill:#e2e3e5,stroke:#333,stroke-width:1px;
    classDef test fill:#f8d7da,stroke:#333,stroke-width:1px;
    classDef note fill:#f0f0f0,stroke-dasharray: 5 5;
```
