How sol0invictus/nanoscaling works
nanoScaling is a research-oriented GPT training and optimizer-comparison repository. Its central workflow configures a training run, loads and batches data, trains a configurable Transformer with a selected optimizer, validates and records diagnostics, and saves checkpoints for later inference or evaluation. Data preparation, supervised fine-tuning, benchmark scoring, and experiment-specific studies are distinct supporting workflows. The README documents several capabilities whose implementations were not sampled; those are represented only where the tree supports a source path, and uncertain call wiring is omitted.
Main entry points: train.py Pretraining loop · gpt.py GPT Transformer · scoring.py Benchmark scoring · train_sft.py SFT training
Components 27
Training
- Pretraining loop
train.py - SFT training
train_sft.py - Configuration
utils/configurator.py - Model checkpoint
Data pipelines
- Batch loading
utils/dataloader.py - Data utilities
utils/data.py - Pretraining data prep
data/openwebtext/prepare.py - SFT data prep
data/prepare_sft.py - Prepared dataset files
Model and optimization
- GPT Transformer
models/gpt.py - Gated DeltaNet
models/gated_delta_net.py - Mixture of Experts
models/moe.py - Muon optimizer
optimizers/muon.py - Scion optimizer
optimizers/scion.py
Evaluation and research
- Spectral diagnostics
utils/spectral_metrics.py - Benchmark scoring
evals/scoring.py - ARC task
evals/tasks/arc.py - GSM8K task
evals/tasks/gsm8k.py - HellaSwag task
evals/tasks/hellaswag.py - MMLU task
evals/tasks/mmlu.py - Bits-per-byte task
evals/tasks/bpb.py - Experiment grids
experiments.py - Spectral geometry study
experiments/rq1_spectral_geometry/train.py - Normalization study
experiments/rq2_normalization/train.py
Inference
- Text generation
inference.py - Sampling entry point
sample.py
Other components
- Researcher
Connections 35
- Researcher → Pretraining loop: starts run
- Researcher → SFT training: starts run
- Researcher → Pretraining data prep: prepares data
- Researcher → SFT data prep: prepares data
- Researcher → Text generation: requests generation
- Pretraining loop → Configuration: loads settings
- SFT training → Configuration: loads settings
- Pretraining loop → Batch loading: loads batches
- SFT training → Batch loading: loads batches
- Batch loading → Data utilities: uses utilities
- Pretraining data prep → Prepared dataset files: writes parquet
- SFT data prep → Prepared dataset files: writes binaries
- Batch loading → Prepared dataset files: reads shards
- Pretraining loop → GPT Transformer: trains model
- SFT training → GPT Transformer: trains model
- GPT Transformer → Gated DeltaNet: optional layers
- GPT Transformer → Mixture of Experts: optional layers
- Pretraining loop → Muon optimizer: selects optimizer
- Pretraining loop → Scion optimizer: selects optimizer
- Pretraining loop → Spectral diagnostics: logs diagnostics
- Spectral geometry study → Spectral diagnostics: records geometry
- Normalization study → Spectral diagnostics: records stability
- Pretraining loop → Model checkpoint: saves model
- SFT training → Model checkpoint: saves model
- Model checkpoint → Text generation: loads weights
- Text generation → GPT Transformer: generates tokens
- Sampling entry point → GPT Transformer: samples output
- ARC task → Benchmark scoring: scores choices
- HellaSwag task → Benchmark scoring: scores choices
- MMLU task → Benchmark scoring: scores choices
- GSM8K task → Benchmark scoring: scores answers
- Bits-per-byte task → Benchmark scoring: scores likelihood
- Benchmark scoring → GPT Transformer: scores model
- Researcher → Experiment grids: runs grid
- Experiment grids → Pretraining loop: configures runs