How test-time-training/ttt-lm-jax works
Project type and purpose
• This is a research‑oriented machine‑learning codebase implemented in JAX for “Learning to (Learn at Test Time),” featuring Test‑Time Training (TTT) layers (linear and MLP) for sequence modeling.
• The goal is to train and benchmark RNN‑style layers with expressive hidden states on large text datasets, leveraging GPUs or Cloud TPUs and logging via Weights & Biases (WandB).
File structure
Top‑level directories and their roles:
• requirements/: dependency specifications for GPU and TPU setups
• scripts/: experiment launch scripts for different model variants (transformer, ttt_linear, ttt_mlp) and model sizes
• ttt/: core Python package
• ttt/dataloader/: data ingestion, tokenization, dataset objects
• ttt/infra/: training infrastructure (checkpointing, optimizer wrappers, JAX utilities)
• ttt/models/: model definitions (base Model class, TTT layers, backprop-through-time module)
• ttt/train.py: training loop and argument parsing
Configuration/build/deployment indicators:
• .gitignore: standard
• requirements/*.txt: pip installable stacks
• scripts/*/*.sh: bash scripts that invoke train.py with the right flags
Key insights from README
• Uses JAX for core computation; tested on GPU/TPU
• Depends on WandB for logging, Google Cloud Storage (gsutil) for dataset download
• Experiment parameters (context length, mesh dimensions) set via command‑line flags in train.py
• References to external repos: PyTorch codebase, kernel implementations, EasyLM and FlashAttention for lineage
a. Main components
• Experiment Launcher (scripts/*.sh)
• CLI Entry Point (ttt/train.py)
• Data Pipeline
• Tokenizer (HF tokenizer folder on disk)
• Dataset class (lm_dataset.py)
• Model Components
• TTT Layer (linear or MLP)
• BPT module (bpt.py)
• Base Model class (model.py)
• Training Infrastructure
• Optimizers (optimizers.py)
• Checkpointing (checkpoint.py)
• JAX utilities (jax_utils.py)
• External Services & Resources
• Compute (GPU or TPU cluster)
• Data storage (GCS buckets)
• Logging (WandB)
b. Relationships & interactions
1. User invokes a shell script → calls Python train.py with flags.
2. train.py initializes WandB, parses config.
3. Data pipeline loads tokenized data from disk via gsutil sync.
4. Batch flows into the Model component, where each step applies TTT layer updates (self‑supervised inner loop) and BPT updates (outer loop).
5. Optimizer updates model weights via JAX on GPU/TPU.
6. Checkpoints written to disk/GCS; metrics sent to WandB.
c. Architectural patterns & principles
• Modular separation (dataloader vs. model vs. infra)
• Experiment-as-code (shell scripts decouple config)
• Functional‑style JAX pipelines for performance
• Extensibility: interchangeable TTT layers, easy addition of new variants
d. Key technologies/frameworks
• JAX (jax.numpy, jit, pmap)
• Weights & Biases (wandb)
• gsutil / Google Cloud Storage
• Bash scripts for reproducible experiments
Detailed breakdown (to capture all essential elements)
• Shell Script Layer
→ Python CLI (train.py)
→ Config Parser → WandB Init
→ DataLoader Module
→ Model Assembly (includes ttt_layer, bpt)
→ Training Loop
• Inner update (self‑supervised on test batch)
• Outer update (optimizer step)
→ Checkpointing & Logging