# test-time-training/ttt-lm-jax architecture

> Architecture diagram of the GitHub repository test-time-training/ttt-lm-jax, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/test-time-training/ttt-lm-jax
- Repository: https://github.com/test-time-training/ttt-lm-jax

## Overview

##### Project type and purpose
- This is a research‑oriented machine‑learning codebase implemented in JAX for “Learning to (Learn at Test Time),” featuring Test‑Time Training (TTT) layers (linear and MLP) for sequence modeling.
- The goal is to train and benchmark RNN‑style layers with expressive hidden states on large text datasets, leveraging GPUs or Cloud TPUs and logging via Weights & Biases (WandB).

##### File structure

Top‑level directories and their roles:
- requirements/: dependency specifications for GPU and TPU setups
- scripts/: experiment launch scripts for different model variants (transformer, ttt_linear, ttt_mlp) and model sizes
- ttt/: core Python package
- ttt/dataloader/: data ingestion, tokenization, dataset objects
- ttt/infra/: training infrastructure (checkpointing, optimizer wrappers, JAX utilities)
- ttt/models/: model definitions (base Model class, TTT layers, backprop-through-time module)
- ttt/train.py: training loop and argument parsing

Configuration/build/deployment indicators:
- .gitignore: standard
- requirements/*.txt: pip installable stacks
- scripts/*/*.sh: bash scripts that invoke train.py with the right flags

##### Key insights from README
- Uses JAX for core computation; tested on GPU/TPU
- Depends on WandB for logging, Google Cloud Storage (gsutil) for dataset download
- Experiment parameters (context length, mesh dimensions) set via command‑line flags in train.py
- References to external repos: PyTorch codebase, kernel implementations, EasyLM and FlashAttention for lineage

a. Main components
- Experiment Launcher (scripts/*.sh)
- CLI Entry Point (ttt/train.py)
- Data Pipeline
- Tokenizer (HF tokenizer folder on disk)
- Dataset class (lm_dataset.py)
- Model Components
- TTT Layer (linear or MLP)
- BPT module (bpt.py)
- Base Model class (model.py)
- Training Infrastructure
- Optimizers (optimizers.py)
- Checkpointing (checkpoint.py)
- JAX utilities (jax_utils.py)
- External Services & Resources
- Compute (GPU or TPU cluster)
- Data storage (GCS buckets)
- Logging (WandB)

b. Relationships & interactions
1. User invokes a shell script → calls Python train.py with flags.
2. train.py initializes WandB, parses config.
3. Data pipeline loads tokenized data from disk via gsutil sync.
4. Batch flows into the Model component, where each step applies TTT layer updates (self‑supervised inner loop) and BPT updates (outer loop).
5. Optimizer updates model weights via JAX on GPU/TPU.
6. Checkpoints written to disk/GCS; metrics sent to WandB.

c. Architectural patterns & principles
- Modular separation (dataloader vs. model vs. infra)
- Experiment-as-code (shell scripts decouple config)
- Functional‑style JAX pipelines for performance
- Extensibility: interchangeable TTT layers, easy addition of new variants

d. Key technologies/frameworks
- JAX (jax.numpy, jit, pmap)
- Weights & Biases (wandb)
- gsutil / Google Cloud Storage
- Bash scripts for reproducible experiments

##### Detailed breakdown (to capture all essential elements)
- Shell Script Layer
→ Python CLI (train.py)
→ Config Parser → WandB Init
→ DataLoader Module
→ Model Assembly (includes ttt_layer, bpt)
→ Training Loop
- Inner update (self‑supervised on test batch)
- Outer update (optimizer step)
→ Checkpointing & Logging

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Experiment Launcher
    subgraph "Experiment Launcher (Shell Scripts)"
        direction TB
        TransformerShells["Transformer Scripts"]:::scripts
        TTTLinearShells["TTT Linear Scripts"]:::scripts
        TTTMLPShells["TTT MLP Scripts"]:::scripts
    end

    %% CLI Entry Point
    TrainCLI["train.py (CLI Entry Point)"]:::cli

    %% Data Pipeline
    subgraph "Data Pipeline" 
        direction TB
        HFLoader["HF Tokenizer Loader"]:::data
        TokenUtils["Tokenization Utilities"]:::data
        Dataset["Dataset Class"]:::data
    end

    %% Model Components
    subgraph "Model Components"
        direction TB
        BaseModel["Base Model Class"]:::model
        TTTLayer["TTT Layer\n(Linear or MLP Variant)"]:::model
        BPT["BPT Module"]:::model
    end

    %% Infrastructure
    subgraph "Training Infrastructure"
        direction TB
        Optimizers["Optimizer Definitions"]:::infra
        Checkpoint["Checkpointing Service"]:::infra
        JAXUtils["JAX Utility Functions"]:::infra
    end

    %% External Services
    subgraph "External Services & Resources"
        direction TB
        Compute["Compute Cluster\n(GPU/TPU)"]:::external
        GCS["GCS Bucket\n(Data Storage)"]:::external
        WandB["Weights & Biases\n(Logging)"]:::external
    end

    %% Connections
    TransformerShells -->|invokes| TrainCLI
    TTTLinearShells -->|invokes| TrainCLI
    TTTMLPShells -->|invokes| TrainCLI

    TrainCLI -->|initializes| WandB
    TrainCLI -->|loads data| HFLoader
    HFLoader --> TokenUtils
    TokenUtils --> Dataset
    GCS -->|sync data| HFLoader

    Dataset -->|batches| BaseModel
    BaseModel --> TTTLayer
    TTTLayer --> BPT
    BPT -->|updates| Optimizers
    Optimizers -->|applies| BaseModel

    Compute --> BaseModel
    Compute --> Optimizers

    Optimizers --> Checkpoint
    Checkpoint -->|writes to| GCS
    BPT -->|logs metrics| WandB

    TrainCLI -->|uses| JAXUtils

    %% Click Events
    click TransformerShells "https://github.com/test-time-training/ttt-lm-jax/tree/main/scripts/transformer"
    click TTTLinearShells "https://github.com/test-time-training/ttt-lm-jax/tree/main/scripts/ttt_linear"
    click TTTMLPShells "https://github.com/test-time-training/ttt-lm-jax/tree/main/scripts/ttt_mlp"
    click TrainCLI "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/train.py"
    click Dataset "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/dataloader/lm_dataset.py"
    click HFLoader "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/dataloader/language_modeling_hf.py"
    click TokenUtils "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/dataloader/tokenization.py"
    click BaseModel "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/models/model.py"
    click TTTLayer "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/models/ttt_layer.py"
    click BPT "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/models/bpt.py"
    click Optimizers "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/infra/optimizers.py"
    click Checkpoint "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/infra/checkpoint.py"
    click JAXUtils "https://github.com/test-time-training/ttt-lm-jax/blob/main/ttt/infra/jax_utils.py"

    %% Styles
    classDef scripts fill:#D8BFD8,stroke:#9370DB,color:#000
    classDef cli fill:#FFE4B5,stroke:#FFA500,color:#000
    classDef data fill:#ADD8E6,stroke:#1E90FF,color:#000
    classDef model fill:#98FB98,stroke:#32CD32,color:#000
    classDef infra fill:#FFD700,stroke:#FFA500,color:#000
    classDef external fill:#D3D3D3,stroke:#A9A9A9,color:#000
```
