# hans0809/minimind-in-depth architecture

> Architecture diagram of the GitHub repository hans0809/minimind-in-depth, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hans0809/minimind-in-depth
- Repository: https://github.com/hans0809/minimind-in-depth

## Overview

MiniMind-in-Depth is a Chinese-language tutorial series for understanding how large language models are built and trained, based on the MiniMind project. The reader’s workflow is to study topics spanning tokenizer construction and model architecture, then pretraining, instruction tuning and DPO, followed by LoRA and distillation. This repository contains explanatory documents rather than an executable training pipeline; the graph therefore shows documented learning topics, not runtime calls. The README establishes the curriculum, but no source excerpts were available to verify finer implementation relationships.

Main entry points: [`src/1-如何从头训练tokenizer.md`](https://github.com/hans0809/minimind-in-depth/blob/main/src/1-%E5%A6%82%E4%BD%95%E4%BB%8E%E5%A4%B4%E8%AE%AD%E7%BB%83tokenizer.md) (Tokenizer training), [`src/2-一行代码之差，模型性能提升背后的RMSNorm玄机.md`](https://github.com/hans0809/minimind-in-depth/blob/main/src/2-%E4%B8%80%E8%A1%8C%E4%BB%A3%E7%A0%81%E4%B9%8B%E5%B7%AE%EF%BC%8C%E6%A8%A1%E5%9E%8B%E6%80%A7%E8%83%BD%E6%8F%90%E5%8D%87%E8%83%8C%E5%90%8E%E7%9A%84RMSNorm%E7%8E%84%E6%9C%BA.md) (RMSNorm), [`src/3-原始Transformer的位置编码及其缺陷.md`](https://github.com/hans0809/minimind-in-depth/blob/main/src/3-%E5%8E%9F%E5%A7%8BTransformer%E7%9A%84%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81%E5%8F%8A%E5%85%B6%E7%BC%BA%E9%99%B7.md) (Position encoding), [`src/4-旋转位置编码原理与应用全解析.md`](https://github.com/hans0809/minimind-in-depth/blob/main/src/4-%E6%97%8B%E8%BD%AC%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81%E5%8E%9F%E7%90%86%E4%B8%8E%E5%BA%94%E7%94%A8%E5%85%A8%E8%A7%A3%E6%9E%90.md) (RoPE)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_foundations["Model foundations"]
  node_tokenizer["Tokenizer training"]
  node_rmsnorm["RMSNorm"]
  node_positional["Position encoding"]
  node_rope["RoPE"]
end

subgraph group_architecture["Model architecture"]
  node_attention["Efficient attention"]
  node_moe["Mixture of experts"]
  node_assembly["Model assembly<br/>[7-像搭积木一样构建一个大模型.md]"]
end

subgraph group_training["Training and alignment"]
  node_pretraining["Pretraining<br/>[8-LLM预训练流程全解.md]"]
  node_instruction["Instruction tuning"]
  node_dpo["DPO alignment"]
end

subgraph group_optimization["Optimization and compression"]
  node_lora["LoRA fine-tuning"]
  node_distillation["Model distillation"]
end

node_reader(("Learner"))

node_reader -->|"studies"| node_tokenizer
node_reader -->|"studies"| node_rmsnorm
node_reader -->|"studies"| node_positional
node_reader -->|"studies"| node_rope
node_reader -->|"studies"| node_attention
node_reader -->|"studies"| node_moe
node_reader -->|"studies"| node_assembly
node_reader -->|"studies"| node_pretraining
node_reader -->|"studies"| node_instruction
node_reader -->|"studies"| node_dpo
node_reader -->|"studies"| node_lora
node_reader -->|"studies"| node_distillation

click node_tokenizer "https://github.com/hans0809/minimind-in-depth/blob/main/src/1-%E5%A6%82%E4%BD%95%E4%BB%8E%E5%A4%B4%E8%AE%AD%E7%BB%83tokenizer.md"
click node_rmsnorm "https://github.com/hans0809/minimind-in-depth/blob/main/src/2-%E4%B8%80%E8%A1%8C%E4%BB%A3%E7%A0%81%E4%B9%8B%E5%B7%AE%EF%BC%8C%E6%A8%A1%E5%9E%8B%E6%80%A7%E8%83%BD%E6%8F%90%E5%8D%87%E8%83%8C%E5%90%8E%E7%9A%84RMSNorm%E7%8E%84%E6%9C%BA.md"
click node_positional "https://github.com/hans0809/minimind-in-depth/blob/main/src/3-%E5%8E%9F%E5%A7%8BTransformer%E7%9A%84%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81%E5%8F%8A%E5%85%B6%E7%BC%BA%E9%99%B7.md"
click node_rope "https://github.com/hans0809/minimind-in-depth/blob/main/src/4-%E6%97%8B%E8%BD%AC%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81%E5%8E%9F%E7%90%86%E4%B8%8E%E5%BA%94%E7%94%A8%E5%85%A8%E8%A7%A3%E6%9E%90.md"
click node_attention "https://github.com/hans0809/minimind-in-depth/blob/main/src/5-%E9%AD%94%E6%94%B9%E7%9A%84%E6%B3%A8%E6%84%8F%E5%8A%9B%E6%9C%BA%E5%88%B6%EF%BC%8C%E7%BB%86%E6%95%B0%E5%BD%93%E4%BB%A3LLM%E7%9A%84%E6%95%88%E7%8E%87%E4%BC%98%E5%8C%96%E6%89%8B%E6%AE%B5.md"
click node_moe "https://github.com/hans0809/minimind-in-depth/blob/main/src/6-%E4%BB%8E%E7%A8%A0%E5%AF%86%E5%88%B0%E7%A8%80%E7%96%8F%EF%BC%8C%E8%AF%A6%E8%A7%A3%E4%B8%93%E5%AE%B6%E6%B7%B7%E5%90%88%E6%A8%A1%E5%9E%8BMOE.md"
click node_assembly "https://github.com/hans0809/minimind-in-depth/blob/main/src/7-%E5%83%8F%E6%90%AD%E7%A7%AF%E6%9C%A8%E4%B8%80%E6%A0%B7%E6%9E%84%E5%BB%BA%E4%B8%80%E4%B8%AA%E5%A4%A7%E6%A8%A1%E5%9E%8B.md"
click node_pretraining "https://github.com/hans0809/minimind-in-depth/blob/main/src/8-LLM%E9%A2%84%E8%AE%AD%E7%BB%83%E6%B5%81%E7%A8%8B%E5%85%A8%E8%A7%A3.md"
click node_instruction "https://github.com/hans0809/minimind-in-depth/blob/main/src/9-%E6%8C%87%E4%BB%A4%E5%BE%AE%E8%B0%83%E8%AF%A6%E8%A7%A3-%E8%AE%A9%E5%A4%A7%E6%A8%A1%E5%9E%8B%E4%BB%8E%E2%80%9C%E8%83%BD%E8%AF%B4%E2%80%9D%E5%8F%98%E5%BE%97%E2%80%9C%E4%BC%9A%E5%90%AC%E2%80%9D.md"
click node_dpo "https://github.com/hans0809/minimind-in-depth/blob/main/src/10-DPO-%E5%A4%A7%E6%A8%A1%E5%9E%8B%E5%AF%B9%E9%BD%90%E8%AE%AD%E7%BB%83%E7%9A%84%E6%96%B0%E8%8C%83%E5%BC%8F.md"
click node_lora "https://github.com/hans0809/minimind-in-depth/blob/main/src/11-LoRA-LLM%E8%BD%BB%E9%87%8F%E5%8C%96%E5%BE%AE%E8%B0%83%E7%9A%84%E5%88%A9%E5%99%A8.md"
click node_distillation "https://github.com/hans0809/minimind-in-depth/blob/main/src/12-%E4%BB%8E%E7%99%BD%E7%9B%92%E5%88%B0%E9%BB%91%E7%9B%92%EF%BC%8C%E5%85%A8%E9%9D%A2%E6%8E%8C%E6%8F%A1%E5%A4%A7%E6%A8%A1%E5%9E%8B%E8%92%B8%E9%A6%8F%E6%8A%80%E6%9C%AF.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tokenizer,node_rmsnorm,node_positional,node_rope toneBlue
class node_attention,node_moe,node_assembly toneAmber
class node_pretraining,node_instruction,node_dpo toneMint
class node_lora,node_distillation toneRose
class node_reader toneIndigo
```

## Components

### Model foundations

- [Tokenizer training](https://github.com/hans0809/minimind-in-depth/blob/main/src/1-%E5%A6%82%E4%BD%95%E4%BB%8E%E5%A4%B4%E8%AE%AD%E7%BB%83tokenizer.md): `src/1-如何从头训练tokenizer.md`
- [RMSNorm](https://github.com/hans0809/minimind-in-depth/blob/main/src/2-%E4%B8%80%E8%A1%8C%E4%BB%A3%E7%A0%81%E4%B9%8B%E5%B7%AE%EF%BC%8C%E6%A8%A1%E5%9E%8B%E6%80%A7%E8%83%BD%E6%8F%90%E5%8D%87%E8%83%8C%E5%90%8E%E7%9A%84RMSNorm%E7%8E%84%E6%9C%BA.md): `src/2-一行代码之差，模型性能提升背后的RMSNorm玄机.md`
- [Position encoding](https://github.com/hans0809/minimind-in-depth/blob/main/src/3-%E5%8E%9F%E5%A7%8BTransformer%E7%9A%84%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81%E5%8F%8A%E5%85%B6%E7%BC%BA%E9%99%B7.md): `src/3-原始Transformer的位置编码及其缺陷.md`
- [RoPE](https://github.com/hans0809/minimind-in-depth/blob/main/src/4-%E6%97%8B%E8%BD%AC%E4%BD%8D%E7%BD%AE%E7%BC%96%E7%A0%81%E5%8E%9F%E7%90%86%E4%B8%8E%E5%BA%94%E7%94%A8%E5%85%A8%E8%A7%A3%E6%9E%90.md): `src/4-旋转位置编码原理与应用全解析.md`

### Model architecture

- [Efficient attention](https://github.com/hans0809/minimind-in-depth/blob/main/src/5-%E9%AD%94%E6%94%B9%E7%9A%84%E6%B3%A8%E6%84%8F%E5%8A%9B%E6%9C%BA%E5%88%B6%EF%BC%8C%E7%BB%86%E6%95%B0%E5%BD%93%E4%BB%A3LLM%E7%9A%84%E6%95%88%E7%8E%87%E4%BC%98%E5%8C%96%E6%89%8B%E6%AE%B5.md): `src/5-魔改的注意力机制，细数当代LLM的效率优化手段.md`
- [Mixture of experts](https://github.com/hans0809/minimind-in-depth/blob/main/src/6-%E4%BB%8E%E7%A8%A0%E5%AF%86%E5%88%B0%E7%A8%80%E7%96%8F%EF%BC%8C%E8%AF%A6%E8%A7%A3%E4%B8%93%E5%AE%B6%E6%B7%B7%E5%90%88%E6%A8%A1%E5%9E%8BMOE.md): `src/6-从稠密到稀疏，详解专家混合模型MOE.md`
- [Model assembly](https://github.com/hans0809/minimind-in-depth/blob/main/src/7-%E5%83%8F%E6%90%AD%E7%A7%AF%E6%9C%A8%E4%B8%80%E6%A0%B7%E6%9E%84%E5%BB%BA%E4%B8%80%E4%B8%AA%E5%A4%A7%E6%A8%A1%E5%9E%8B.md): `src/7-像搭积木一样构建一个大模型.md`

### Training and alignment

- [Pretraining](https://github.com/hans0809/minimind-in-depth/blob/main/src/8-LLM%E9%A2%84%E8%AE%AD%E7%BB%83%E6%B5%81%E7%A8%8B%E5%85%A8%E8%A7%A3.md): `src/8-LLM预训练流程全解.md`
- [Instruction tuning](https://github.com/hans0809/minimind-in-depth/blob/main/src/9-%E6%8C%87%E4%BB%A4%E5%BE%AE%E8%B0%83%E8%AF%A6%E8%A7%A3-%E8%AE%A9%E5%A4%A7%E6%A8%A1%E5%9E%8B%E4%BB%8E%E2%80%9C%E8%83%BD%E8%AF%B4%E2%80%9D%E5%8F%98%E5%BE%97%E2%80%9C%E4%BC%9A%E5%90%AC%E2%80%9D.md): `src/9-指令微调详解-让大模型从“能说”变得“会听”.md`
- [DPO alignment](https://github.com/hans0809/minimind-in-depth/blob/main/src/10-DPO-%E5%A4%A7%E6%A8%A1%E5%9E%8B%E5%AF%B9%E9%BD%90%E8%AE%AD%E7%BB%83%E7%9A%84%E6%96%B0%E8%8C%83%E5%BC%8F.md): `src/10-DPO-大模型对齐训练的新范式.md`

### Optimization and compression

- [LoRA fine-tuning](https://github.com/hans0809/minimind-in-depth/blob/main/src/11-LoRA-LLM%E8%BD%BB%E9%87%8F%E5%8C%96%E5%BE%AE%E8%B0%83%E7%9A%84%E5%88%A9%E5%99%A8.md): `src/11-LoRA-LLM轻量化微调的利器.md`
- [Model distillation](https://github.com/hans0809/minimind-in-depth/blob/main/src/12-%E4%BB%8E%E7%99%BD%E7%9B%92%E5%88%B0%E9%BB%91%E7%9B%92%EF%BC%8C%E5%85%A8%E9%9D%A2%E6%8E%8C%E6%8F%A1%E5%A4%A7%E6%A8%A1%E5%9E%8B%E8%92%B8%E9%A6%8F%E6%8A%80%E6%9C%AF.md): `src/12-从白盒到黑盒，全面掌握大模型蒸馏技术.md`

### Other components

- **Learner**

## Connections

- Learner → Tokenizer training: studies
- Learner → RMSNorm: studies
- Learner → Position encoding: studies
- Learner → RoPE: studies
- Learner → Efficient attention: studies
- Learner → Mixture of experts: studies
- Learner → Model assembly: studies
- Learner → Pretraining: studies
- Learner → Instruction tuning: studies
- Learner → DPO alignment: studies
- Learner → LoRA fine-tuning: studies
- Learner → Model distillation: studies