# fareedkhan-dev/train-llm-from-scratch architecture

> Architecture diagram of the GitHub repository fareedkhan-dev/train-llm-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fareedkhan-dev/train-llm-from-scratch
- Repository: https://github.com/fareedkhan-dev/train-llm-from-scratch

## Overview

Train LLM From Scratch is a PyTorch toolkit and control panel for taking text from corpus preparation through pretraining, alignment, evaluation, and chat. The README documents the principal progression; source excerpts additionally show checkpoint-based inference, chat formatting, and the UI job manager. The model internals and several training scripts are not fully sampled, so those links rely on the README’s documented workflow rather than inferred calls.

Main entry points: [`scripts/pretrain_base.py`](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/scripts/pretrain_base.py) (Base pretraining), [`src/post_training/inference.py`](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/inference.py) (Checkpoint inference), [`ui/app.py`](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/ui/app.py) (Streamlit control panel), [`src/post_training/sft.py`](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/sft.py) (Supervised fine-tuning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_prep["Corpus preparation"]
  node_loader["Dataset loading<br/>[data_loader.py]"]
  node_tokenizer["BPE tokenizer<br/>[bpe.py]"]
end

subgraph group_models["Models and runtime"]
  node_classic["Classic Transformer<br/>[transformer.py]"]
  node_modern["Modern Transformer<br/>[model.py]"]
  node_modern_block["Modern blocks<br/>[block.py]"]
  node_modern_attention["Modern attention<br/>[attention.py]"]
  node_factory["Model factory<br/>[factory.py]"]
  node_checkpoint["Checkpoint handling<br/>[checkpoint.py]"]
end

subgraph group_training["Training and alignment"]
  node_pretrain["Base pretraining<br/>[pretrain_base.py]"]
  node_sft["Supervised fine-tuning<br/>[sft.py]"]
  node_reward["Reward modeling<br/>[reward_train.py]"]
  node_preference["Preference optimization<br/>[dpo.py]"]
  node_ppo["PPO training<br/>[ppo.py]"]
  node_grpo["GRPO training<br/>[grpo.py]"]
end

subgraph group_serving["Evaluation and inference"]
  node_chat_template["Chat formatting<br/>[chat_template.py]"]
  node_sampling["Token sampling<br/>[sampling.py]"]
  node_inference["Checkpoint inference<br/>[inference.py]"]
  node_evaluation["Evaluation<br/>[evaluation.py]"]
end

subgraph group_control["Control panel"]
  node_ui["Streamlit control panel<br/>[app.py]"]
  node_stages["Stage registry<br/>[stages.py]"]
  node_jobs["Background jobs<br/>[jobs.py]"]
end

node_user(("Engineer"))

node_user -->|"prepares data"| node_prep
node_prep -->|"produces datasets"| node_loader
node_loader -.->|"encodes text"| node_tokenizer
node_tokenizer -->|"provides tokens"| node_pretrain
node_pretrain -->|"trains model"| node_classic
node_pretrain -->|"trains model"| node_modern
node_modern -->|"runs blocks"| node_modern_block
node_modern_block -->|"uses attention"| node_modern_attention
node_factory -->|"builds model"| node_classic
node_factory -->|"builds model"| node_modern
node_pretrain -->|"saves checkpoints"| node_checkpoint
node_checkpoint -->|"loads weights"| node_inference
node_pretrain -->|"fine-tunes base"| node_sft
node_sft -->|"trains reward model"| node_reward
node_reward -->|"supports PPO"| node_ppo
node_sft -->|"optimizes preferences"| node_preference
node_preference -->|"continues alignment"| node_grpo
node_inference -->|"builds model"| node_factory
node_inference -->|"loads checkpoint"| node_checkpoint
node_inference -->|"formats prompts"| node_chat_template
node_inference -->|"generates replies"| node_evaluation
node_evaluation -.->|"samples tokens"| node_sampling
node_user -->|"launches workflow"| node_ui
node_ui -->|"manages jobs"| node_jobs
node_jobs -->|"reads stages"| node_stages
node_stages -->|"configures stage"| node_pretrain
node_ui -->|"offers evaluation"| node_evaluation
node_ui -->|"offers chat"| node_inference

click node_prep "https://github.com/fareedkhan-dev/train-llm-from-scratch/tree/main/scripts"
click node_loader "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/data_loader/data_loader.py"
click node_tokenizer "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/tokenizer/bpe.py"
click node_classic "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py"
click node_modern "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/modern/model.py"
click node_modern_block "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/modern/block.py"
click node_modern_attention "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/modern/attention.py"
click node_factory "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/factory.py"
click node_checkpoint "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/checkpoint.py"
click node_pretrain "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/scripts/pretrain_base.py"
click node_sft "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/sft.py"
click node_reward "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_train.py"
click node_preference "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/dpo.py"
click node_ppo "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/ppo.py"
click node_grpo "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/grpo.py"
click node_chat_template "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/chat_template.py"
click node_sampling "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/inference/sampling.py"
click node_inference "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/inference.py"
click node_evaluation "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/evaluation.py"
click node_ui "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/ui/app.py"
click node_stages "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/ui/stages.py"
click node_jobs "https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/ui/jobs.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prep,node_loader,node_tokenizer toneBlue
class node_classic,node_modern,node_modern_block,node_modern_attention,node_factory,node_checkpoint toneAmber
class node_pretrain,node_sft,node_reward,node_preference,node_ppo,node_grpo toneMint
class node_chat_template,node_sampling,node_inference,node_evaluation toneRose
class node_ui,node_stages,node_jobs,node_user toneIndigo
```

## Components

### Data preparation

- [Corpus preparation](https://github.com/fareedkhan-dev/train-llm-from-scratch/tree/main/scripts): `scripts`
- [Dataset loading](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/data_loader/data_loader.py): `data_loader/data_loader.py`
- [BPE tokenizer](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/tokenizer/bpe.py): `src/tokenizer/bpe.py`

### Models and runtime

- [Classic Transformer](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py): `src/models/transformer.py`
- [Modern Transformer](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/modern/model.py): `src/models/modern/model.py`
- [Modern blocks](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/modern/block.py): `src/models/modern/block.py`
- [Modern attention](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/modern/attention.py): `src/models/modern/attention.py`
- [Model factory](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/models/factory.py): `src/models/factory.py`
- [Checkpoint handling](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/checkpoint.py): `src/checkpoint.py`

### Training and alignment

- [Base pretraining](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/scripts/pretrain_base.py): `scripts/pretrain_base.py`
- [Supervised fine-tuning](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/sft.py): `src/post_training/sft.py`
- [Reward modeling](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_train.py): `src/post_training/reward_train.py`
- [Preference optimization](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/dpo.py): `src/post_training/dpo.py`
- [PPO training](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/ppo.py): `src/post_training/ppo.py`
- [GRPO training](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/grpo.py): `src/post_training/grpo.py`

### Evaluation and inference

- [Chat formatting](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/chat_template.py): `src/post_training/chat_template.py`
- [Token sampling](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/inference/sampling.py): `src/inference/sampling.py`
- [Checkpoint inference](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/inference.py): `src/post_training/inference.py`
- [Evaluation](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/src/post_training/evaluation.py): `src/post_training/evaluation.py`

### Control panel

- [Streamlit control panel](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/ui/app.py): `ui/app.py`
- [Stage registry](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/ui/stages.py): `ui/stages.py`
- [Background jobs](https://github.com/fareedkhan-dev/train-llm-from-scratch/blob/main/ui/jobs.py): `ui/jobs.py`

### Other components

- **Engineer**

## Connections

- Engineer → Corpus preparation: prepares data (evidence: `README.md`)
- Corpus preparation → Dataset loading: produces datasets (evidence: `README.md`)
- Dataset loading → BPE tokenizer: encodes text
- BPE tokenizer → Base pretraining: provides tokens (evidence: `README.md`)
- Base pretraining → Classic Transformer: trains model (evidence: `README.md`)
- Base pretraining → Modern Transformer: trains model (evidence: `README.md`)
- Modern Transformer → Modern blocks: runs blocks (evidence: `src/models/modern/model.py`)
- Modern blocks → Modern attention: uses attention (evidence: `src/models/modern/block.py`)
- Model factory → Classic Transformer: builds model (evidence: `src/models/factory.py`)
- Model factory → Modern Transformer: builds model (evidence: `src/models/factory.py`)
- Base pretraining → Checkpoint handling: saves checkpoints (evidence: `README.md`)
- Checkpoint handling → Checkpoint inference: loads weights (evidence: `src/post_training/inference.py`)
- Base pretraining → Supervised fine-tuning: fine-tunes base (evidence: `README.md`)
- Supervised fine-tuning → Reward modeling: trains reward model (evidence: `README.md`)
- Reward modeling → PPO training: supports PPO (evidence: `README.md`)
- Supervised fine-tuning → Preference optimization: optimizes preferences (evidence: `README.md`)
- Preference optimization → GRPO training: continues alignment (evidence: `README.md`)
- Checkpoint inference → Model factory: builds model (evidence: `src/post_training/inference.py`)
- Checkpoint inference → Checkpoint handling: loads checkpoint (evidence: `src/post_training/inference.py`)
- Checkpoint inference → Chat formatting: formats prompts (evidence: `src/post_training/inference.py`)
- Checkpoint inference → Evaluation: generates replies (evidence: `src/post_training/inference.py`)
- Evaluation → Token sampling: samples tokens
- Engineer → Streamlit control panel: launches workflow (evidence: `README.md`)
- Streamlit control panel → Background jobs: manages jobs (evidence: `ui/app.py`)
- Background jobs → Stage registry: reads stages (evidence: `ui/jobs.py`)
- Stage registry → Base pretraining: configures stage (evidence: `ui/stages.py`)
- Streamlit control panel → Evaluation: offers evaluation (evidence: `README.md`)
- Streamlit control panel → Checkpoint inference: offers chat
