# sol0invictus/nanoscaling architecture

> Architecture diagram of the GitHub repository sol0invictus/nanoscaling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sol0invictus/nanoscaling
- Repository: https://github.com/sol0invictus/nanoscaling

## Overview

nanoScaling is a research-oriented GPT training and optimizer-comparison repository. Its central workflow configures a training run, loads and batches data, trains a configurable Transformer with a selected optimizer, validates and records diagnostics, and saves checkpoints for later inference or evaluation. Data preparation, supervised fine-tuning, benchmark scoring, and experiment-specific studies are distinct supporting workflows. The README documents several capabilities whose implementations were not sampled; those are represented only where the tree supports a source path, and uncertain call wiring is omitted.

Main entry points: [`train.py`](https://github.com/sol0invictus/nanoscaling/blob/main/train.py) (Pretraining loop), [`models/gpt.py`](https://github.com/sol0invictus/nanoscaling/blob/main/models/gpt.py) (GPT Transformer), [`evals/scoring.py`](https://github.com/sol0invictus/nanoscaling/blob/main/evals/scoring.py) (Benchmark scoring), [`train_sft.py`](https://github.com/sol0invictus/nanoscaling/blob/main/train_sft.py) (SFT training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training"]
  node_pretrain["Pretraining loop<br/>[train.py]"]
  node_sft_train["SFT training<br/>[train_sft.py]"]
  node_config["Configuration<br/>[configurator.py]"]
  node_checkpoint["Model checkpoint"]
end

subgraph group_data["Data pipelines"]
  node_batching["Batch loading<br/>[dataloader.py]"]
  node_data_utils["Data utilities<br/>[data.py]"]
  node_pretrain_prep["Pretraining data prep<br/>[prepare.py]"]
  node_sft_prep["SFT data prep<br/>[prepare_sft.py]"]
  node_dataset_files["Prepared dataset files"]
end

subgraph group_model["Model and optimization"]
  node_gpt["GPT Transformer<br/>[gpt.py]"]
  node_gated_delta["Gated DeltaNet<br/>[gated_delta_net.py]"]
  node_moe["Mixture of Experts<br/>[moe.py]"]
  node_muon["Muon optimizer<br/>[muon.py]"]
  node_scion["Scion optimizer<br/>[scion.py]"]
end

subgraph group_evaluation["Evaluation and research"]
  node_metrics["Spectral diagnostics"]
  node_task_scoring["Benchmark scoring<br/>[scoring.py]"]
  node_task_arc["ARC task<br/>[arc.py]"]
  node_task_gsm["GSM8K task<br/>[gsm8k.py]"]
  node_task_hella["HellaSwag task<br/>[hellaswag.py]"]
  node_task_mmlu["MMLU task<br/>[mmlu.py]"]
  node_task_bpb["Bits-per-byte task<br/>[bpb.py]"]
  node_experiment_runner["Experiment grids<br/>[experiments.py]"]
  node_rq1["Spectral geometry study<br/>[train.py]"]
  node_rq2["Normalization study<br/>[train.py]"]
end

subgraph group_inference["Inference"]
  node_inference_cli["Text generation<br/>[inference.py]"]
  node_sample_cli["Sampling entry point<br/>[sample.py]"]
end

node_researcher(("Researcher"))

node_researcher -->|"starts run"| node_pretrain
node_researcher -->|"starts run"| node_sft_train
node_researcher -->|"prepares data"| node_pretrain_prep
node_researcher -->|"prepares data"| node_sft_prep
node_researcher -->|"requests generation"| node_inference_cli
node_pretrain -->|"loads settings"| node_config
node_sft_train -->|"loads settings"| node_config
node_pretrain -->|"loads batches"| node_batching
node_sft_train -->|"loads batches"| node_batching
node_batching -->|"uses utilities"| node_data_utils
node_pretrain_prep -->|"writes parquet"| node_dataset_files
node_sft_prep -->|"writes binaries"| node_dataset_files
node_batching -->|"reads shards"| node_dataset_files
node_pretrain -->|"trains model"| node_gpt
node_sft_train -->|"trains model"| node_gpt
node_gpt -.->|"optional layers"| node_gated_delta
node_gpt -.->|"optional layers"| node_moe
node_pretrain -.->|"selects optimizer"| node_muon
node_pretrain -.->|"selects optimizer"| node_scion
node_pretrain -->|"logs diagnostics"| node_metrics
node_rq1 -->|"records geometry"| node_metrics
node_rq2 -->|"records stability"| node_metrics
node_pretrain -->|"saves model"| node_checkpoint
node_sft_train -->|"saves model"| node_checkpoint
node_checkpoint -->|"loads weights"| node_inference_cli
node_inference_cli -->|"generates tokens"| node_gpt
node_sample_cli -->|"samples output"| node_gpt
node_task_arc -->|"scores choices"| node_task_scoring
node_task_hella -->|"scores choices"| node_task_scoring
node_task_mmlu -->|"scores choices"| node_task_scoring
node_task_gsm -->|"scores answers"| node_task_scoring
node_task_bpb -->|"scores likelihood"| node_task_scoring
node_task_scoring -->|"scores model"| node_gpt
node_researcher -->|"runs grid"| node_experiment_runner
node_experiment_runner -->|"configures runs"| node_pretrain

click node_pretrain "https://github.com/sol0invictus/nanoscaling/blob/main/train.py"
click node_sft_train "https://github.com/sol0invictus/nanoscaling/blob/main/train_sft.py"
click node_config "https://github.com/sol0invictus/nanoscaling/blob/main/utils/configurator.py"
click node_batching "https://github.com/sol0invictus/nanoscaling/blob/main/utils/dataloader.py"
click node_data_utils "https://github.com/sol0invictus/nanoscaling/blob/main/utils/data.py"
click node_pretrain_prep "https://github.com/sol0invictus/nanoscaling/blob/main/data/openwebtext/prepare.py"
click node_sft_prep "https://github.com/sol0invictus/nanoscaling/blob/main/data/prepare_sft.py"
click node_gpt "https://github.com/sol0invictus/nanoscaling/blob/main/models/gpt.py"
click node_gated_delta "https://github.com/sol0invictus/nanoscaling/blob/main/models/gated_delta_net.py"
click node_moe "https://github.com/sol0invictus/nanoscaling/blob/main/models/moe.py"
click node_muon "https://github.com/sol0invictus/nanoscaling/blob/main/optimizers/muon.py"
click node_scion "https://github.com/sol0invictus/nanoscaling/blob/main/optimizers/scion.py"
click node_metrics "https://github.com/sol0invictus/nanoscaling/blob/main/utils/spectral_metrics.py"
click node_task_scoring "https://github.com/sol0invictus/nanoscaling/blob/main/evals/scoring.py"
click node_task_arc "https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/arc.py"
click node_task_gsm "https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/gsm8k.py"
click node_task_hella "https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/hellaswag.py"
click node_task_mmlu "https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/mmlu.py"
click node_task_bpb "https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/bpb.py"
click node_experiment_runner "https://github.com/sol0invictus/nanoscaling/blob/main/experiments.py"
click node_rq1 "https://github.com/sol0invictus/nanoscaling/blob/main/experiments/rq1_spectral_geometry/train.py"
click node_rq2 "https://github.com/sol0invictus/nanoscaling/blob/main/experiments/rq2_normalization/train.py"
click node_inference_cli "https://github.com/sol0invictus/nanoscaling/blob/main/inference.py"
click node_sample_cli "https://github.com/sol0invictus/nanoscaling/blob/main/sample.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pretrain,node_sft_train,node_config,node_checkpoint toneBlue
class node_batching,node_data_utils,node_pretrain_prep,node_sft_prep,node_dataset_files toneAmber
class node_gpt,node_gated_delta,node_moe,node_muon,node_scion toneMint
class node_metrics,node_task_scoring,node_task_arc,node_task_gsm,node_task_hella,node_task_mmlu,node_task_bpb,node_experiment_runner,node_rq1,node_rq2 toneRose
class node_inference_cli,node_sample_cli,node_researcher toneIndigo
```

## Components

### Training

- [Pretraining loop](https://github.com/sol0invictus/nanoscaling/blob/main/train.py): `train.py`
- [SFT training](https://github.com/sol0invictus/nanoscaling/blob/main/train_sft.py): `train_sft.py`
- [Configuration](https://github.com/sol0invictus/nanoscaling/blob/main/utils/configurator.py): `utils/configurator.py`
- **Model checkpoint**

### Data pipelines

- [Batch loading](https://github.com/sol0invictus/nanoscaling/blob/main/utils/dataloader.py): `utils/dataloader.py`
- [Data utilities](https://github.com/sol0invictus/nanoscaling/blob/main/utils/data.py): `utils/data.py`
- [Pretraining data prep](https://github.com/sol0invictus/nanoscaling/blob/main/data/openwebtext/prepare.py): `data/openwebtext/prepare.py`
- [SFT data prep](https://github.com/sol0invictus/nanoscaling/blob/main/data/prepare_sft.py): `data/prepare_sft.py`
- **Prepared dataset files**

### Model and optimization

- [GPT Transformer](https://github.com/sol0invictus/nanoscaling/blob/main/models/gpt.py): `models/gpt.py`
- [Gated DeltaNet](https://github.com/sol0invictus/nanoscaling/blob/main/models/gated_delta_net.py): `models/gated_delta_net.py`
- [Mixture of Experts](https://github.com/sol0invictus/nanoscaling/blob/main/models/moe.py): `models/moe.py`
- [Muon optimizer](https://github.com/sol0invictus/nanoscaling/blob/main/optimizers/muon.py): `optimizers/muon.py`
- [Scion optimizer](https://github.com/sol0invictus/nanoscaling/blob/main/optimizers/scion.py): `optimizers/scion.py`

### Evaluation and research

- [Spectral diagnostics](https://github.com/sol0invictus/nanoscaling/blob/main/utils/spectral_metrics.py): `utils/spectral_metrics.py`
- [Benchmark scoring](https://github.com/sol0invictus/nanoscaling/blob/main/evals/scoring.py): `evals/scoring.py`
- [ARC task](https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/arc.py): `evals/tasks/arc.py`
- [GSM8K task](https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/gsm8k.py): `evals/tasks/gsm8k.py`
- [HellaSwag task](https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/hellaswag.py): `evals/tasks/hellaswag.py`
- [MMLU task](https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/mmlu.py): `evals/tasks/mmlu.py`
- [Bits-per-byte task](https://github.com/sol0invictus/nanoscaling/blob/main/evals/tasks/bpb.py): `evals/tasks/bpb.py`
- [Experiment grids](https://github.com/sol0invictus/nanoscaling/blob/main/experiments.py): `experiments.py`
- [Spectral geometry study](https://github.com/sol0invictus/nanoscaling/blob/main/experiments/rq1_spectral_geometry/train.py): `experiments/rq1_spectral_geometry/train.py`
- [Normalization study](https://github.com/sol0invictus/nanoscaling/blob/main/experiments/rq2_normalization/train.py): `experiments/rq2_normalization/train.py`

### Inference

- [Text generation](https://github.com/sol0invictus/nanoscaling/blob/main/inference.py): `inference.py`
- [Sampling entry point](https://github.com/sol0invictus/nanoscaling/blob/main/sample.py): `sample.py`

### Other components

- **Researcher**

## Connections

- Researcher → Pretraining loop: starts run
- Researcher → SFT training: starts run
- Researcher → Pretraining data prep: prepares data
- Researcher → SFT data prep: prepares data
- Researcher → Text generation: requests generation
- Pretraining loop → Configuration: loads settings
- SFT training → Configuration: loads settings
- Pretraining loop → Batch loading: loads batches
- SFT training → Batch loading: loads batches
- Batch loading → Data utilities: uses utilities
- Pretraining data prep → Prepared dataset files: writes parquet
- SFT data prep → Prepared dataset files: writes binaries
- Batch loading → Prepared dataset files: reads shards
- Pretraining loop → GPT Transformer: trains model
- SFT training → GPT Transformer: trains model
- GPT Transformer → Gated DeltaNet: optional layers
- GPT Transformer → Mixture of Experts: optional layers
- Pretraining loop → Muon optimizer: selects optimizer
- Pretraining loop → Scion optimizer: selects optimizer
- Pretraining loop → Spectral diagnostics: logs diagnostics
- Spectral geometry study → Spectral diagnostics: records geometry
- Normalization study → Spectral diagnostics: records stability
- Pretraining loop → Model checkpoint: saves model
- SFT training → Model checkpoint: saves model
- Model checkpoint → Text generation: loads weights
- Text generation → GPT Transformer: generates tokens
- Sampling entry point → GPT Transformer: samples output
- ARC task → Benchmark scoring: scores choices
- HellaSwag task → Benchmark scoring: scores choices
- MMLU task → Benchmark scoring: scores choices
- GSM8K task → Benchmark scoring: scores answers
- Bits-per-byte task → Benchmark scoring: scores likelihood
- Benchmark scoring → GPT Transformer: scores model
- Researcher → Experiment grids: runs grid
- Experiment grids → Pretraining loop: configures runs