# skyzh/tiny-llm architecture

> Architecture diagram of the GitHub repository skyzh/tiny-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/skyzh/tiny-llm
- Repository: https://github.com/skyzh/tiny-llm

## Overview

tiny-llm is both a hands-on LLM-inference course and a compact Qwen3 serving implementation. Its main user workflow loads a model, dispatches it through a week-specific implementation, and generates sampled text; later serving stages add batching and paged KV attention. A separate Week 4 path runs a bounded coding agent over a workspace, with checkpointing and evidence-oriented capabilities. The graph combines README-documented stages with sampled imports and calls; several serving and agent features are represented from documentation or indexed files rather than inspected implementation excerpts.

Main entry points: [`src/tiny_llm/qwen3_week2.py`](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/qwen3_week2.py) (Qwen3 Model), [`src/tiny_llm/agent/loop.py`](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/loop.py) (Agent Loop), [`main.py`](https://github.com/skyzh/tiny-llm/blob/main/main.py) (Inference CLI), [`src/tiny_llm/agent/workspace.py`](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/workspace.py) (Workspace Tools)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inference["Model Inference"]
  node_model_registry["Model Registry<br/>[model_names.py]"]
  node_model_loader["Model Loader<br/>[models.py]"]
  node_qwen_model["Qwen3 Model<br/>[qwen3_week2.py]"]
  node_attention["Attention<br/>[attention.py]"]
  node_quantization["Quantization<br/>[quantize.py]"]
  node_embedding["Embedding<br/>[embedding.py]"]
end

subgraph group_serving["Serving Runtime"]
  node_kv_cache[("KV Cache<br/>[kv_cache.py]")]
  node_batching["Batch Scheduler<br/>[batch.py]"]
  node_paged_cache[("Paged KV Cache<br/>[paged_kv_cache.py]")]
  node_generation["Generation Loop<br/>[generate.py]"]
  node_sampler["Sampler<br/>[sampler.py]"]
  node_cli["Inference CLI<br/>[main.py]"]
end

subgraph group_kernels["Kernel Implementations"]
  node_native_kernels["Native Kernels<br/>[tiny_llm_ext.h]"]
  node_reference_kernels["Reference Kernels<br/>[tiny_llm_ext.h]"]
end

subgraph group_agent["Coding Agent"]
  node_agent_cli["Agent Entry<br/>[agent.py]"]
  node_agent_loop["Agent Loop<br/>[loop.py]"]
  node_agent_protocol["Action Protocol<br/>[protocol.py]"]
  node_workspace["Workspace Tools<br/>[workspace.py]"]
  node_checkpoint["Checkpointing<br/>[checkpoint.py]"]
  node_receipts[("Effect Receipts<br/>[receipts.py]")]
  node_compaction["Context Compaction<br/>[compaction.py]"]
  node_steering["Agent Steering<br/>[steering.py]"]
  node_branching["Branch Selection<br/>[branching.py]"]
  node_evaluation["Run Evaluation<br/>[evaluation.py]"]
  node_evidence["Tool Evidence<br/>[evidence.py]"]
end

node_user(("User"))

node_user -->|"prompts"| node_cli
node_cli -->|"resolves model"| node_model_registry
node_cli -->|"dispatches model"| node_model_loader
node_model_loader -->|"selects implementation"| node_qwen_model
node_qwen_model -->|"uses attention"| node_attention
node_qwen_model -->|"uses weights"| node_quantization
node_qwen_model -.->|"updates cache"| node_kv_cache
node_batching -->|"uses cache"| node_kv_cache
node_kv_cache -->|"supports paging"| node_paged_cache
node_qwen_model -.->|"uses embeddings"| node_embedding
node_cli -->|"runs generation"| node_generation
node_generation -.->|"samples tokens"| node_sampler
node_qwen_model -.->|"dispatches kernels"| node_native_kernels
node_qwen_model -.->|"dispatches kernels"| node_reference_kernels
node_paged_cache -.->|"updates pages"| node_reference_kernels
node_user -->|"starts task"| node_agent_cli
node_agent_cli -.->|"runs agent"| node_agent_loop
node_agent_loop -->|"validates actions"| node_agent_protocol
node_agent_loop -.->|"executes actions"| node_workspace
node_agent_loop -->|"saves boundary"| node_checkpoint
node_workspace -->|"records effects"| node_receipts
node_agent_loop -.->|"compacts context"| node_compaction
node_agent_loop -.->|"accepts steering"| node_steering
node_agent_loop -.->|"forks continuations"| node_branching
node_branching -.->|"evaluates outcomes"| node_evaluation
node_workspace -->|"stores tool results"| node_evidence

click node_model_registry "https://github.com/skyzh/tiny-llm/blob/main/model_names.py"
click node_model_loader "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/models.py"
click node_qwen_model "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/qwen3_week2.py"
click node_attention "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/attention.py"
click node_quantization "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/quantize.py"
click node_embedding "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/embedding.py"
click node_kv_cache "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/kv_cache.py"
click node_batching "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/batch.py"
click node_paged_cache "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/paged_kv_cache.py"
click node_generation "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/generate.py"
click node_sampler "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/sampler.py"
click node_cli "https://github.com/skyzh/tiny-llm/blob/main/main.py"
click node_native_kernels "https://github.com/skyzh/tiny-llm/blob/main/src/extensions/src/tiny_llm_ext.h"
click node_reference_kernels "https://github.com/skyzh/tiny-llm/blob/main/src/extensions_ref/src/tiny_llm_ext.h"
click node_agent_cli "https://github.com/skyzh/tiny-llm/blob/main/agent.py"
click node_agent_loop "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/loop.py"
click node_agent_protocol "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/protocol.py"
click node_workspace "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/workspace.py"
click node_checkpoint "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/checkpoint.py"
click node_receipts "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/receipts.py"
click node_compaction "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/compaction.py"
click node_steering "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/steering.py"
click node_branching "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/branching.py"
click node_evaluation "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/evaluation.py"
click node_evidence "https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/evidence.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_model_registry,node_model_loader,node_qwen_model,node_attention,node_quantization,node_embedding,node_user toneBlue
class node_kv_cache,node_batching,node_paged_cache,node_generation,node_sampler,node_cli toneAmber
class node_native_kernels,node_reference_kernels toneMint
class node_agent_cli,node_agent_loop,node_agent_protocol,node_workspace,node_checkpoint,node_receipts,node_compaction,node_steering,node_branching,node_evaluation,node_evidence toneRose
```

## Components

### Model Inference

- [Model Registry](https://github.com/skyzh/tiny-llm/blob/main/model_names.py): `model_names.py`
- [Model Loader](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/models.py): `src/tiny_llm/models.py`
- [Qwen3 Model](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/qwen3_week2.py): `src/tiny_llm/qwen3_week2.py`
- [Attention](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/attention.py): `src/tiny_llm/attention.py`
- [Quantization](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/quantize.py): `src/tiny_llm/quantize.py`
- [Embedding](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/embedding.py): `src/tiny_llm/embedding.py`

### Serving Runtime

- [KV Cache](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/kv_cache.py): `src/tiny_llm/kv_cache.py`
- [Batch Scheduler](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/batch.py): `src/tiny_llm/batch.py`
- [Paged KV Cache](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/paged_kv_cache.py): `src/tiny_llm/paged_kv_cache.py`
- [Generation Loop](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/generate.py): `src/tiny_llm/generate.py`
- [Sampler](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/sampler.py): `src/tiny_llm/sampler.py`
- [Inference CLI](https://github.com/skyzh/tiny-llm/blob/main/main.py): `main.py`

### Kernel Implementations

- [Native Kernels](https://github.com/skyzh/tiny-llm/blob/main/src/extensions/src/tiny_llm_ext.h): `src/extensions/src/tiny_llm_ext.h`
- [Reference Kernels](https://github.com/skyzh/tiny-llm/blob/main/src/extensions_ref/src/tiny_llm_ext.h): `src/extensions_ref/src/tiny_llm_ext.h`

### Coding Agent

- [Agent Entry](https://github.com/skyzh/tiny-llm/blob/main/agent.py): `agent.py`
- [Agent Loop](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/loop.py): `src/tiny_llm/agent/loop.py`
- [Action Protocol](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/protocol.py): `src/tiny_llm/agent/protocol.py`
- [Workspace Tools](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/workspace.py): `src/tiny_llm/agent/workspace.py`
- [Checkpointing](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/checkpoint.py): `src/tiny_llm/agent/checkpoint.py`
- [Effect Receipts](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/receipts.py): `src/tiny_llm/agent/receipts.py`
- [Context Compaction](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/compaction.py): `src/tiny_llm/agent/compaction.py`
- [Agent Steering](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/steering.py): `src/tiny_llm/agent/steering.py`
- [Branch Selection](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/branching.py): `src/tiny_llm/agent/branching.py`
- [Run Evaluation](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/evaluation.py): `src/tiny_llm/agent/evaluation.py`
- [Tool Evidence](https://github.com/skyzh/tiny-llm/blob/main/src/tiny_llm/agent/evidence.py): `src/tiny_llm/agent/evidence.py`

### Other components

- **User**

## Connections

- User → Inference CLI: prompts (evidence: `README.md`)
- Inference CLI → Model Registry: resolves model (evidence: `main.py`)
- Inference CLI → Model Loader: dispatches model (evidence: `main.py`)
- Model Loader → Qwen3 Model: selects implementation (evidence: `src/tiny_llm/models.py`)
- Qwen3 Model → Attention: uses attention
- Qwen3 Model → Quantization: uses weights
- Qwen3 Model → KV Cache: updates cache
- Batch Scheduler → KV Cache: uses cache (evidence: `src/tiny_llm/batch.py`)
- KV Cache → Paged KV Cache: supports paging (evidence: `src/tiny_llm/kv_cache.py`)
- Qwen3 Model → Embedding: uses embeddings
- Inference CLI → Generation Loop: runs generation (evidence: `main.py`)
- Generation Loop → Sampler: samples tokens
- Qwen3 Model → Native Kernels: dispatches kernels
- Qwen3 Model → Reference Kernels: dispatches kernels
- Paged KV Cache → Reference Kernels: updates pages
- User → Agent Entry: starts task (evidence: `README.md`)
- Agent Entry → Agent Loop: runs agent
- Agent Loop → Action Protocol: validates actions (evidence: `src/tiny_llm/agent/loop.py`)
- Agent Loop → Workspace Tools: executes actions
- Agent Loop → Checkpointing: saves boundary (evidence: `src/tiny_llm/agent/loop.py`)
- Workspace Tools → Effect Receipts: records effects (evidence: `src/tiny_llm/agent/workspace.py`)
- Agent Loop → Context Compaction: compacts context
- Agent Loop → Agent Steering: accepts steering
- Agent Loop → Branch Selection: forks continuations
- Branch Selection → Run Evaluation: evaluates outcomes
- Workspace Tools → Tool Evidence: stores tool results
