How amankumarranjan8969/slm- works
SelfLearningLLM is a small-language-model project with a data-to-training workflow: train or load a tokenizer, pack a corpus into token shards, sample training windows, and train a decoder-only model with checkpoints. The sampled runtime code supports these core stages and model generation; main.py describes additional commands, but its excerpt says implementations are not yet available. The README provides no further capability detail, so unsampled subsystems are not expanded here.
Main entry points: trainer.py Training loop · transformer.py Transformer stack · packing.py Corpus packer · llm.py Causal LM
Components 21
Entry and data
- Unified CLI
SelfLearningLLM/main.py - Experiment config
SelfLearningLLM/model/config.py - Text corpus
SelfLearningLLM/data/raw - Corpus packer
SelfLearningLLM/data/packing.py - Packed token shards
SelfLearningLLM/data
Tokenizer
- BPE trainer
SelfLearningLLM/tokenizer/train_tokenizer.py - Tokenizer files
SelfLearningLLM/tokenizer - Tokenizer API
SelfLearningLLM/tokenizer/tokenizer.py
Training
- Packed dataset
SelfLearningLLM/training/dataset.py - Training loop
SelfLearningLLM/training/trainer.py - Optimizer
SelfLearningLLM/training/optimizer.py - LR scheduler
SelfLearningLLM/training/scheduler.py - Training checkpoints
SelfLearningLLM/training/checkpoint.py
Model
- Causal LM
SelfLearningLLM/model/llm.py - Token embedding
SelfLearningLLM/model/embeddings.py - Transformer stack
SelfLearningLLM/model/transformer.py - Grouped attention
SelfLearningLLM/model/attention.py - Rotary embeddings
SelfLearningLLM/model/rope.py - SwiGLU feedforward
SelfLearningLLM/model/feedforward.py - RMS normalization
SelfLearningLLM/model/normalization.py
Other components
- Operator
Connections 20
- Operator → Unified CLI: runs commands
- Unified CLI → Experiment config: loads config
- BPE trainer → Text corpus: reads text
- BPE trainer → Tokenizer files: writes artifacts
- Tokenizer API → Tokenizer files: loads artifacts
- Corpus packer → Text corpus: reads documents
- Corpus packer → Tokenizer API: encodes text
- Corpus packer → Packed token shards: writes tokens
- Packed dataset → Packed token shards: reads windows
- Training loop → Packed dataset: loads batches
- Training loop → Causal LM: runs forward
- Training loop → Optimizer: updates weights
- Training loop → LR scheduler: sets learning rate
- Training loop → Training checkpoints: saves state
- Causal LM → Token embedding: embeds tokens
- Causal LM → Transformer stack: runs stack
- Transformer stack → Grouped attention: applies attention
- Transformer stack → SwiGLU feedforward: applies feedforward
- Transformer stack → RMS normalization: normalizes layers
- Grouped attention → Rotary embeddings: applies rotations