# amankumarranjan8969/slm- architecture

> Architecture diagram of the GitHub repository amankumarranjan8969/slm-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amankumarranjan8969/slm-
- Repository: https://github.com/amankumarranjan8969/slm-

## Overview

SelfLearningLLM is a small-language-model project with a data-to-training workflow: train or load a tokenizer, pack a corpus into token shards, sample training windows, and train a decoder-only model with checkpoints. The sampled runtime code supports these core stages and model generation; `main.py` describes additional commands, but its excerpt says implementations are not yet available. The README provides no further capability detail, so unsampled subsystems are not expanded here.

Main entry points: [`SelfLearningLLM/training/trainer.py`](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/trainer.py) (Training loop), [`SelfLearningLLM/model/transformer.py`](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/transformer.py) (Transformer stack), [`SelfLearningLLM/data/packing.py`](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/data/packing.py) (Corpus packer), [`SelfLearningLLM/model/llm.py`](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/llm.py) (Causal LM)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry_data["Entry and data"]
  node_cli["Unified CLI<br/>[main.py]"]
  node_config["Experiment config<br/>[config.py]"]
  node_corpus["Text corpus"]
  node_packer["Corpus packer<br/>[packing.py]"]
  node_shards[("Packed token shards")]
end

subgraph group_tokenizer["Tokenizer"]
  node_tokenizer_train["BPE trainer<br/>[train_tokenizer.py]"]
  node_tokenizer_files["Tokenizer files"]
  node_tokenizer_api["Tokenizer API<br/>[tokenizer.py]"]
end

subgraph group_training["Training"]
  node_dataset["Packed dataset<br/>[dataset.py]"]
  node_trainer["Training loop<br/>[trainer.py]"]
  node_optimizer["Optimizer<br/>[optimizer.py]"]
  node_scheduler["LR scheduler<br/>[scheduler.py]"]
  node_checkpoints[("Training checkpoints<br/>[checkpoint.py]")]
end

subgraph group_model["Model"]
  node_lm["Causal LM<br/>[llm.py]"]
  node_embedding["Token embedding<br/>[embeddings.py]"]
  node_transformer["Transformer stack<br/>[transformer.py]"]
  node_attention["Grouped attention<br/>[attention.py]"]
  node_rope["Rotary embeddings<br/>[rope.py]"]
  node_feedforward["SwiGLU feedforward<br/>[feedforward.py]"]
  node_normalization["RMS normalization<br/>[normalization.py]"]
end

node_user(("Operator"))

node_user -->|"runs commands"| node_cli
node_cli -->|"loads config"| node_config
node_tokenizer_train -->|"reads text"| node_corpus
node_tokenizer_train -->|"writes artifacts"| node_tokenizer_files
node_tokenizer_api -->|"loads artifacts"| node_tokenizer_files
node_packer -->|"reads documents"| node_corpus
node_packer -->|"encodes text"| node_tokenizer_api
node_packer -->|"writes tokens"| node_shards
node_dataset -->|"reads windows"| node_shards
node_trainer -->|"loads batches"| node_dataset
node_trainer -->|"runs forward"| node_lm
node_trainer -->|"updates weights"| node_optimizer
node_trainer -->|"sets learning rate"| node_scheduler
node_trainer -->|"saves state"| node_checkpoints
node_lm -->|"embeds tokens"| node_embedding
node_lm -->|"runs stack"| node_transformer
node_transformer -->|"applies attention"| node_attention
node_transformer -->|"applies feedforward"| node_feedforward
node_transformer -->|"normalizes layers"| node_normalization
node_attention -->|"applies rotations"| node_rope

click node_cli "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/main.py"
click node_config "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/config.py"
click node_corpus "https://github.com/amankumarranjan8969/slm-/tree/main/SelfLearningLLM/data/raw"
click node_packer "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/data/packing.py"
click node_shards "https://github.com/amankumarranjan8969/slm-/tree/main/SelfLearningLLM/data"
click node_tokenizer_train "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/tokenizer/train_tokenizer.py"
click node_tokenizer_files "https://github.com/amankumarranjan8969/slm-/tree/main/SelfLearningLLM/tokenizer"
click node_tokenizer_api "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/tokenizer/tokenizer.py"
click node_dataset "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/dataset.py"
click node_trainer "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/trainer.py"
click node_optimizer "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/optimizer.py"
click node_scheduler "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/scheduler.py"
click node_checkpoints "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/checkpoint.py"
click node_lm "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/llm.py"
click node_embedding "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/embeddings.py"
click node_transformer "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/transformer.py"
click node_attention "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/attention.py"
click node_rope "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/rope.py"
click node_feedforward "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/feedforward.py"
click node_normalization "https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/normalization.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_corpus,node_packer,node_shards toneBlue
class node_tokenizer_train,node_tokenizer_files,node_tokenizer_api toneAmber
class node_dataset,node_trainer,node_optimizer,node_scheduler,node_checkpoints toneMint
class node_lm,node_embedding,node_transformer,node_attention,node_rope,node_feedforward,node_normalization toneRose
class node_user toneIndigo
```

## Components

### Entry and data

- [Unified CLI](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/main.py): `SelfLearningLLM/main.py`
- [Experiment config](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/config.py): `SelfLearningLLM/model/config.py`
- [Text corpus](https://github.com/amankumarranjan8969/slm-/tree/main/SelfLearningLLM/data/raw): `SelfLearningLLM/data/raw`
- [Corpus packer](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/data/packing.py): `SelfLearningLLM/data/packing.py`
- [Packed token shards](https://github.com/amankumarranjan8969/slm-/tree/main/SelfLearningLLM/data): `SelfLearningLLM/data`

### Tokenizer

- [BPE trainer](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/tokenizer/train_tokenizer.py): `SelfLearningLLM/tokenizer/train_tokenizer.py`
- [Tokenizer files](https://github.com/amankumarranjan8969/slm-/tree/main/SelfLearningLLM/tokenizer): `SelfLearningLLM/tokenizer`
- [Tokenizer API](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/tokenizer/tokenizer.py): `SelfLearningLLM/tokenizer/tokenizer.py`

### Training

- [Packed dataset](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/dataset.py): `SelfLearningLLM/training/dataset.py`
- [Training loop](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/trainer.py): `SelfLearningLLM/training/trainer.py`
- [Optimizer](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/optimizer.py): `SelfLearningLLM/training/optimizer.py`
- [LR scheduler](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/scheduler.py): `SelfLearningLLM/training/scheduler.py`
- [Training checkpoints](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/training/checkpoint.py): `SelfLearningLLM/training/checkpoint.py`

### Model

- [Causal LM](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/llm.py): `SelfLearningLLM/model/llm.py`
- [Token embedding](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/embeddings.py): `SelfLearningLLM/model/embeddings.py`
- [Transformer stack](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/transformer.py): `SelfLearningLLM/model/transformer.py`
- [Grouped attention](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/attention.py): `SelfLearningLLM/model/attention.py`
- [Rotary embeddings](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/rope.py): `SelfLearningLLM/model/rope.py`
- [SwiGLU feedforward](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/feedforward.py): `SelfLearningLLM/model/feedforward.py`
- [RMS normalization](https://github.com/amankumarranjan8969/slm-/blob/main/SelfLearningLLM/model/normalization.py): `SelfLearningLLM/model/normalization.py`

### Other components

- **Operator**

## Connections

- Operator → Unified CLI: runs commands
- Unified CLI → Experiment config: loads config
- BPE trainer → Text corpus: reads text
- BPE trainer → Tokenizer files: writes artifacts
- Tokenizer API → Tokenizer files: loads artifacts
- Corpus packer → Text corpus: reads documents
- Corpus packer → Tokenizer API: encodes text
- Corpus packer → Packed token shards: writes tokens
- Packed dataset → Packed token shards: reads windows
- Training loop → Packed dataset: loads batches
- Training loop → Causal LM: runs forward
- Training loop → Optimizer: updates weights
- Training loop → LR scheduler: sets learning rate
- Training loop → Training checkpoints: saves state
- Causal LM → Token embedding: embeds tokens
- Causal LM → Transformer stack: runs stack
- Transformer stack → Grouped attention: applies attention
- Transformer stack → SwiGLU feedforward: applies feedforward
- Transformer stack → RMS normalization: normalizes layers
- Grouped attention → Rotary embeddings: applies rotations