How jainilpatel2502/minireasoner-70m works
Project type and purpose
• This is a monolithic Python transformer modeling project (educational full-stack pipeline).
• Purpose: implement, train, and serve a 70M-parameter transformer from scratch—including data ingestion, tokenizer training, model definition, training loop, checkpointing/logging, and inference scripts.
File structure
Top-level modules:
• MultiheadAttention.py, RMSNorm.py, SwiGLU.py, rope.py, rotatory_pos_embd.py
– low-level building blocks for attention, normalization, activations, positional embeddings
• TransformerBlock.py, Transformer.py
– assemble blocks into full model architecture
• training.py
– orchestrates data loading, forward/backward passes, optimizer, LR schedule, checkpointing, mixed precision
• sample_outputs.py
– inference script to generate text samples from a checkpoint
• app.py
– lightweight serving or interactive prompt wrapper around the model
• count_par.py
– utility to inspect model parameter counts
Data/ directory:
• data_collector_for_tokenization.py, tokenizer.py, DataLoader.py, data_fetcher.py, test_tokenizer.py
– build and exercise a custom tokenizer, streaming DataLoader that reads synthetic dataset
• tokenizer.model, tokenizer.vocab
– artifacts produced by tokenizer training
Extract README insights
• Tech stack: PyTorch, FlashAttention, bfloat16 mixed precision, TensorBoard
• Features: PreNorm design, SwiGLU, RMSNorm, RoPE, weight-tying, cosine LR decay, checkpoint every 500 steps
• Data origin: nvidia/OpenCodeReasoning synthetic tokens (external dataset)
Main components and interactions
A. Data Ingestion & Tokenizer Training
– External dataset (Hugging Face / local files) → data_fetcher.py → data_collector_for_tokenization.py
– tokenizer.py ← trains BPE/WordPiece model → outputs tokenizer.model & tokenizer.vocab
B. DataLoader
– streaming loader (DataLoader.py) uses tokenizer to convert raw text → token IDs in batches
C. Model Definition
– building blocks (MultiheadAttention, RMSNorm, SwiGLU, RoPE) → TransformerBlock → Transformer
D. Training Engine
– training.py:
· loads DataLoader batches → feeds into Transformer
· applies optimizer, gradient clipping, mixed precision (bfloat16)
· updates weights, logs metrics to TensorBoard
· checkpointing module saves model & optimizer states every N steps
E. Utilities
– count_par.py inspects total parameters
F. Inference / Serving
– sample_outputs.py: loads checkpoint, runs generate() with sampling settings
– app.py: wraps sampling in an interactive script or API endpoint