How skyzh/tiny-llm works
tiny-llm is both a hands-on LLM-inference course and a compact Qwen3 serving implementation. Its main user workflow loads a model, dispatches it through a week-specific implementation, and generates sampled text; later serving stages add batching and paged KV attention. A separate Week 4 path runs a bounded coding agent over a workspace, with checkpointing and evidence-oriented capabilities. The graph combines README-documented stages with sampled imports and calls; several serving and agent features are represented from documentation or indexed files rather than inspected implementation excerpts.
Main entry points: qwen3_week2.py Qwen3 Model · loop.py Agent Loop · main.py Inference CLI · workspace.py Workspace Tools
Components 26
Model Inference
- Model Registry
model_names.py - Model Loader
src/tiny_llm/models.py - Qwen3 Model
src/tiny_llm/qwen3_week2.py - Attention
src/tiny_llm/attention.py - Quantization
src/tiny_llm/quantize.py - Embedding
src/tiny_llm/embedding.py
Serving Runtime
- KV Cache
src/tiny_llm/kv_cache.py - Batch Scheduler
src/tiny_llm/batch.py - Paged KV Cache
src/tiny_llm/paged_kv_cache.py - Generation Loop
src/tiny_llm/generate.py - Sampler
src/tiny_llm/sampler.py - Inference CLI
main.py
Kernel Implementations
- Native Kernels
src/extensions/src/tiny_llm_ext.h - Reference Kernels
src/extensions_ref/src/tiny_llm_ext.h
Coding Agent
- Agent Entry
agent.py - Agent Loop
src/tiny_llm/agent/loop.py - Action Protocol
src/tiny_llm/agent/protocol.py - Workspace Tools
src/tiny_llm/agent/workspace.py - Checkpointing
src/tiny_llm/agent/checkpoint.py - Effect Receipts
src/tiny_llm/agent/receipts.py - Context Compaction
src/tiny_llm/agent/compaction.py - Agent Steering
src/tiny_llm/agent/steering.py - Branch Selection
src/tiny_llm/agent/branching.py - Run Evaluation
src/tiny_llm/agent/evaluation.py - Tool Evidence
src/tiny_llm/agent/evidence.py
Other components
- User
Connections 26
- User → Inference CLI: prompts
README.md - Inference CLI → Model Registry: resolves model
main.py - Inference CLI → Model Loader: dispatches model
main.py - Model Loader → Qwen3 Model: selects implementation
src/tiny_llm/models.py - Qwen3 Model → Attention: uses attention
- Qwen3 Model → Quantization: uses weights
- Qwen3 Model → KV Cache: updates cache
- Batch Scheduler → KV Cache: uses cache
src/tiny_llm/batch.py - KV Cache → Paged KV Cache: supports paging
src/tiny_llm/kv_cache.py - Qwen3 Model → Embedding: uses embeddings
- Inference CLI → Generation Loop: runs generation
main.py - Generation Loop → Sampler: samples tokens
- Qwen3 Model → Native Kernels: dispatches kernels
- Qwen3 Model → Reference Kernels: dispatches kernels
- Paged KV Cache → Reference Kernels: updates pages
- User → Agent Entry: starts task
README.md - Agent Entry → Agent Loop: runs agent
- Agent Loop → Action Protocol: validates actions
src/tiny_llm/agent/loop.py - Agent Loop → Workspace Tools: executes actions
- Agent Loop → Checkpointing: saves boundary
src/tiny_llm/agent/loop.py - Workspace Tools → Effect Receipts: records effects
src/tiny_llm/agent/workspace.py - Agent Loop → Context Compaction: compacts context
- Agent Loop → Agent Steering: accepts steering
- Agent Loop → Branch Selection: forks continuations
- Branch Selection → Run Evaluation: evaluates outcomes
- Workspace Tools → Tool Evidence: stores tool results