# nihilisticneuralnet/tinyrecursivemodels architecture

> Architecture diagram of the GitHub repository nihilisticneuralnet/tinyrecursivemodels, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nihilisticneuralnet/tinyrecursivemodels
- Repository: https://github.com/nihilisticneuralnet/tinyrecursivemodels

## Overview

##### Project type and purpose
- This is an ML research pipeline implementing the Tiny Recursion Model (TRM) to solve ARC-AGI and puzzle tasks.
- Key features: recursive reasoning loops, tiny parameter count (≈7M), configurable via YAML, distributed training on GPUs, logging to Weights & Biases.

##### Top-level components
A. Configuration Management
- config/arch/*.yaml (architecture presets: trm, hrm, transformers_baseline, etc.)
- config/cfg_pretrain.yaml (training hyperparameters)
– Role: Hydra-style config loader that parameterizes model size, cycles, learning rates, data paths.
B. Data Ingestion & Preprocessing
- dataset/*.py (build_arc_dataset, build_sudoku_dataset, build_maze_dataset, common.py)
- kaggle/combined/*.json (raw ARC-AGI challenge & solution sets)
– Role: generate training/evaluation sets, augment data, produce torch Datasets.
C. Model Definition
- models/common.py (embeddings, shared utils)
- models/layers.py, sparse_embedding.py, ema.py, losses.py
- models/recursive_reasoning/… (trm.py, trm_hier6.py, hrm.py, etc.)
– Role: implement Tiny Recursion architecture: embedding, latent z recursion loops, answer updates.
D. Training Orchestrator
- pretrain.py (entry point for training jobs)
- puzzle_dataset.py (wrapper for puzzle tasks)
– Role: initialize distributed PyTorch, load config, build datasets and model, run training loops, call evaluators.
E. Evaluation & Metrics
- evaluators/arc.py (compute ARC metrics)
- optional in-loop evaluators (passed to pretrain.py)
– Role: evaluate model snapshots on held-out tasks, produce accuracy metrics.
F. Utilities & Logging
- utils/functions.py (misc helper functions)
- external service: Weights & Biases (wandb) for experiment tracking
– Role: logging loss curves, metrics, saving checkpoints via wandb API.

##### Relationships & Data Flow
1. pretrain.py
– loads config files → builds DataLoader via dataset/*.py → instantiates Model via models/… → enters training loop.
2. Training loop
– For each batch: forward pass (embedding → recursive reasoning cycles → answer update) → compute loss → backward → optimizer step → EMA update.
– Periodically call evaluators/arc.py or other evaluators → log metrics to wandb → checkpoint model.
3. Config files feed into both data loading parameters and model architecture parameters.
4. kaggle JSONs are raw inputs; dataset build scripts convert them into ready-to-use torch datasets.

##### Architectural Patterns & Design Principles
- Modular separation: config, data, model, training, evaluation, utils.
- Recursion loop: latent “z” update repeated H times, alternating with answer “y” updates L times.
- Hydra-style config injection: decoupling code from hyperparameters.
- PyTorch Distributed Data Parallel (DDP) for multi-GPU.

##### Final checklist
- All major code modules (dataset, models, pretrain, evaluators, utils).
- Config layer and how it injects parameters.
- Data sources (kaggle → build scripts → DataLoader).
- Model internals: latent recursion and answer update.
- Training control flow & distributed execution.
- Evaluation / metrics flow → wandb.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Config Layer
    subgraph "Configuration Management"
        Config["Configuration Management"]:::config
        Arch["Architecture Presets (config/arch/*.yaml)"]:::config
        CfgPre["Training Hyperparameters (config/cfg_pretrain.yaml)"]:::config
    end

    %% Data Layer
    subgraph "Data Ingestion & Preprocessing"
        RawData["Raw Data Sources (kaggle/combined/*.json)"]:::data
        BuildArc["build_arc_dataset.py"]:::code
        BuildSudoku["build_sudoku_dataset.py"]:::code
        BuildMaze["build_maze_dataset.py"]:::code
        CommonBuild["common.py"]:::code
    end

    %% Model Layer
    subgraph "Model Definition"
        CommonUtils["models/common.py"]:::code
        Layers["models/layers.py"]:::code
        SparseEmb["models/sparse_embedding.py"]:::code
        EMA["models/ema.py"]:::code
        Losses["models/losses.py"]:::code
        subgraph "Recursive Reasoning Variants"
            TRM["trm.py"]:::code
            TRMH6["trm_hier6.py"]:::code
            SingleZ["trm_singlez.py"]:::code
            HRM["hrm.py"]:::code
            TBase["transformers_baseline.py"]:::code
        end
    end

    %% Training Orchestrator
    subgraph "Training"
        Entry["pretrain.py"]:::code
        PuzzleWrap["puzzle_dataset.py"]:::code
        TrainingLoop["Training Loop"]:::process
        Dependencies["requirements.txt"]:::code
    end

    %% Evaluation & Metrics
    subgraph "Evaluation & Metrics"
        ArcEval["evaluators/arc.py"]:::code
    end

    %% Utilities
    Utils["utils/functions.py"]:::code

    %% External Services
    subgraph "External Services"
        WAndB["Weights & Biases"]:::external
        GPU["GPU Cluster (DDP)"]:::external
    end

    %% Flows
    Config -->|load config| Entry
    Arch --> Entry
    CfgPre --> Entry

    RawData -->|raw input| BuildArc
    RawData -->|raw input| BuildSudoku
    RawData -->|raw input| BuildMaze
    BuildArc -->|produce dataset| TrainingLoop
    BuildSudoku --> TrainingLoop
    BuildMaze --> TrainingLoop
    CommonBuild --> BuildArc
    CommonBuild --> BuildSudoku
    CommonBuild --> BuildMaze

    Entry -->|initialize| TrainingLoop
    Entry -->|instantiate| CommonUtils
    Entry --> PuzzleWrap
    TrainingLoop -->|uses| CommonUtils
    TrainingLoop -->|forward/backward| Layers
    TrainingLoop -->|embed| SparseEmb
    TrainingLoop -->|update EMA| EMA
    TrainingLoop -->|compute loss| Losses
    TrainingLoop -->|call model variants| TRM
    TrainingLoop --> TRMH6
    TrainingLoop --> SingleZ
    TrainingLoop --> HRM
    TrainingLoop --> TBase

    TrainingLoop -->|distributed| GPU
    TrainingLoop -->|periodic eval| ArcEval
    ArcEval -->|metrics| WAndB
    TrainingLoop -->|log metrics| WAndB

    Utils --> Entry
    Dependencies --> Entry

    %% Click Events
    click Config "https://github.com/nihilisticneuralnet/tinyrecursivemodels/tree/main/config/"
    click Arch "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/config/arch/*.yaml"
    click CfgPre "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/config/cfg_pretrain.yaml"
    click BuildArc "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/dataset/build_arc_dataset.py"
    click BuildSudoku "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/dataset/build_sudoku_dataset.py"
    click BuildMaze "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/dataset/build_maze_dataset.py"
    click CommonBuild "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/dataset/common.py"
    click RawData "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/kaggle/combined/*.json"
    click CommonUtils "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/common.py"
    click Layers "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/layers.py"
    click SparseEmb "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/sparse_embedding.py"
    click EMA "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/ema.py"
    click Losses "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/losses.py"
    click TRM "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/recursive_reasoning/trm.py"
    click TRMH6 "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/recursive_reasoning/trm_hier6.py"
    click SingleZ "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/recursive_reasoning/trm_singlez.py"
    click HRM "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/recursive_reasoning/hrm.py"
    click TBase "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/models/recursive_reasoning/transformers_baseline.py"
    click Entry "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/pretrain.py"
    click PuzzleWrap "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/puzzle_dataset.py"
    click ArcEval "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/evaluators/arc.py"
    click Utils "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/utils/functions.py"
    click Dependencies "https://github.com/nihilisticneuralnet/tinyrecursivemodels/blob/main/requirements.txt"

    %% Styles
    classDef config fill:#FAD390,stroke:#E59866
    classDef data fill:#82E0AA,stroke:#229954
    classDef code fill:#74B9FF,stroke:#0984E3
    classDef external fill:#A29BFE,stroke:#6C5CE7
    classDef process fill:#FFEAA7,stroke:#FAB1A0
```
