How nihilisticneuralnet/tinyrecursivemodels works
Project type and purpose
• This is an ML research pipeline implementing the Tiny Recursion Model (TRM) to solve ARC-AGI and puzzle tasks.
• Key features: recursive reasoning loops, tiny parameter count (≈7M), configurable via YAML, distributed training on GPUs, logging to Weights & Biases.
Top-level components
A. Configuration Management
• config/arch/*.yaml (architecture presets: trm, hrm, transformers_baseline, etc.)
• config/cfg_pretrain.yaml (training hyperparameters)
– Role: Hydra-style config loader that parameterizes model size, cycles, learning rates, data paths.
B. Data Ingestion & Preprocessing
• dataset/*.py (build_arc_dataset, build_sudoku_dataset, build_maze_dataset, common.py)
• kaggle/combined/*.json (raw ARC-AGI challenge & solution sets)
– Role: generate training/evaluation sets, augment data, produce torch Datasets.
C. Model Definition
• models/common.py (embeddings, shared utils)
• models/layers.py, sparse_embedding.py, ema.py, losses.py
• models/recursive_reasoning/… (trm.py, trm_hier6.py, hrm.py, etc.)
– Role: implement Tiny Recursion architecture: embedding, latent z recursion loops, answer updates.
D. Training Orchestrator
• pretrain.py (entry point for training jobs)
• puzzle_dataset.py (wrapper for puzzle tasks)
– Role: initialize distributed PyTorch, load config, build datasets and model, run training loops, call evaluators.
E. Evaluation & Metrics
• evaluators/arc.py (compute ARC metrics)
• optional in-loop evaluators (passed to pretrain.py)
– Role: evaluate model snapshots on held-out tasks, produce accuracy metrics.
F. Utilities & Logging
• utils/functions.py (misc helper functions)
• external service: Weights & Biases (wandb) for experiment tracking
– Role: logging loss curves, metrics, saving checkpoints via wandb API.
Relationships & Data Flow
1. pretrain.py
– loads config files → builds DataLoader via dataset/*.py → instantiates Model via models/… → enters training loop.
2. Training loop
– For each batch: forward pass (embedding → recursive reasoning cycles → answer update) → compute loss → backward → optimizer step → EMA update.
– Periodically call evaluators/arc.py or other evaluators → log metrics to wandb → checkpoint model.
3. Config files feed into both data loading parameters and model architecture parameters.
4. kaggle JSONs are raw inputs; dataset build scripts convert them into ready-to-use torch datasets.
Architectural Patterns & Design Principles
• Modular separation: config, data, model, training, evaluation, utils.
• Recursion loop: latent “z” update repeated H times, alternating with answer “y” updates L times.
• Hydra-style config injection: decoupling code from hyperparameters.
• PyTorch Distributed Data Parallel (DDP) for multi-GPU.
Final checklist
• All major code modules (dataset, models, pretrain, evaluators, utils).
• Config layer and how it injects parameters.
• Data sources (kaggle → build scripts → DataLoader).
• Model internals: latent recursion and answer update.
• Training control flow & distributed execution.
• Evaluation / metrics flow → wandb.