How avalokkumar/learning-llm works
1. This repository is an educational LLM curriculum, not a single application. Its architecture is organized as a linear learning pipeline: each week builds on the previous one, moving from text preprocessing to transformer internals, training, alignment, retrieval, fine-tuning, and inference optimization.
2. The primary unit of composition is the Jupyter notebook. Each weekX/dayY folder is a self-contained lesson with one or more notebooks and a short README.md that frames the topic. For an architecture diagram, these notebooks should be treated as modular learning/workflow nodes rather than runtime services.
3. Week 1 is the text-processing foundation. It covers normalization, tokenization, embeddings, positional encodings, and a full preprocessing pipeline. The only visible code-first helpers live here: week1/day1/bert_tokenizer.py and tokenizer_examples.py, which support the tokenization lessons and reinforce the boundary between raw text and model-ready tokens.
4. Week 2 is the transformer core. The notebooks progress through scaled dot-product attention, multi-head attention, transformer variants, component layers such as residuals and layer norm, and then a minimal end-to-end transformer. This is the main model-architecture subsystem in the repo.
5. Week 3 focuses on training data and pretraining. It includes dataset/perplexity work, tokenization at scale, sequence packing, the training loop, regularization/monitoring, and a nanoGPT-style pretraining notebook. This week is the bridge from “model components” to “trainable language model,” with practical dataflow from text corpora into tokenized training sequences.
6. Week 3 also contains concrete artifacts that act as inputs to later notebooks: week3/day12/data/*.txt provides sample corpora, and week3/day12/tokenizers/*.json stores trained tokenizer vocabularies for BPE, Unigram, and WordPiece. These are the main persistent data assets in the repo.
7. Week 4 is the alignment and evaluation layer. It covers supervised fine-tuning, RLHF components, DPO, safety implementation, and evaluation. Architecturally, this is the post-pretraining adaptation boundary: base model outputs are refined with instruction data, preference data, safety rules, and benchmark-driven evaluation.
8. Week 5 shifts to prompting and retrieval-augmented generation. The notebooks move from prompt engineering to RAG, advanced retrieval, RAG optimization, and RAG evaluation/safety. This introduces an external-knowledge path: queries are augmented with retrieved context before generation, rather than relying only on the parametric model.
9. Week 6 is parameter-efficient adaptation and compression-oriented training. It includes LoRA, QLoRA, distillation, catastrophic forgetting, hyperparameter tuning, early stopping, adapter merging, and model cards. This subsystem emphasizes efficient model specialization and model governance artifacts.
10. Week 7 covers inference-time optimization and serving. The notebooks address quantization, KV caching, flash attention, paged attention, batching strategies, scheduling, local model serving, streaming APIs, TGI, TensorRT-LLM, autoscaling, caching, rate limiting, multi-tenancy, metrics, logging, tracing, GPU monitoring, and telemetry. For a diagram, this is the deployment/runtime boundary where trained models become serviced workloads.
11. The diagrams/ folder is the visual support layer for the curriculum. It is not the core learning path, but it likely contains reference graphics used to explain model internals and system relationships.
12. .github/workflows/python-package-conda.yml is the only explicit automation boundary visible. It suggests basic CI/environment validation around the Python/conda setup, supporting reproducibility for notebook execution rather than deploying an application.
13. Overall data flow in the repo is instructional: raw text and corpora feed tokenizers; tokenized sequences feed embeddings and transformers; trained base models feed fine-tuning, preference optimization, and safety work; then retrieval, quantization, batching, and serving turn the models into usable systems. The architecture is best drawn as a staged curriculum with shared assets and artifacts crossing week boundaries.
Main entry points: day1_tokenization_practical.ipynb Text Prep · day12_tokenization_part1.ipynb Data & Tokenization · day10_minimal_transformer_part1.ipynb Mini Transformer · README.md LLM Curriculum
Components 24
Curriculum
Main week-by-week learning pipeline
- LLM Curriculum · Top-level educational path across all weeks
README.md - Text Prep · Normalization and tokenization lessons
week1/day1/day1_tokenization_practical.ipynb - Embeddings · Embeddings and positional encodings
week1/day3/day3_embeddings_part1.ipynb - Preprocess Pipeline · End-to-end text-to-model-ready pipeline
week1/day5/day5_pipeline_part1.ipynb - Attention · Scaled dot-product and multi-head attention
week2/day6/day6_attention_implementation.ipynb - Transformer Blocks · Residuals, layer norm, and transformer variants
week2/day9/day9_transformer_components_part1.ipynb - Mini Transformer · Minimal end-to-end transformer model
week2/day10/day10_minimal_transformer_part1.ipynb - Data & Tokenization · Datasets, corpora, and tokenizer assets
week3/day12/day12_tokenization_part1.ipynb - Sequence Packing · Packing tokenized examples for training
week3/day12/day12_sequence_packing.ipynb - Training Loop · Core training loop and optimization flow
week3/day13/day13_training_loop_complete.ipynb - Pretraining · NanoGPT-style pretraining notebook
week3/day15/day15_nanogpt_pretraining.ipynb - Alignment · Supervised fine-tuning, RLHF, and DPO
week4/day16/day16_supervised_fine_tuning.ipynb - Safety & Eval · Safety implementation and benchmark evaluation
week4/day19/day19_safety_implementation.ipynb - Prompting & RAG · Prompt engineering and retrieval-augmented generation
week5/day22/day22_rag_implementation.ipynb - Retrieval · Advanced retrieval and RAG optimization
week5/day23/day23_advanced_retrieval.ipynb - PEFT · LoRA, QLoRA, and adapter-based specialization
week6/day26/day26_lora_implementation_part1.ipynb - Distillation · Distillation, forgetting, tuning, and model governance
week6/day28/day28_distillation_implementation_part1.ipynb - Inference · Quantization and decoding-time acceleration
week7/day31/day31_int8_quantization_part1.ipynb - Serving Stack · Local serving, APIs, batching, and deployment
week7/day33/day33_local_model_server_part1.ipynb - Ops & Telemetry · Caching, rate limiting, multitenancy, logging, and monitoring
week7/day34/day34_caching_strategies_part1.ipynb
Shared Assets
Persistent data, helper code, and visual references
- Tokenizer Helpers · Code-first tokenizer boundary for lessons
week1/day1/bert_tokenizer.py - Corpus Assets · Sample text corpora and trained tokenizer vocabularies
week3/day12/data/all_texts.txt - Diagrams · Reference graphics for the curriculum
diagrams
Support
Repository automation and reproducibility
- CI Workflow · Python/conda validation and environment setup
.github/workflows/python-package-conda.yml
Connections 23
- LLM Curriculum → Text Prep: starts with · Curriculum begins with text preprocessing and tokenization
- Text Prep → Tokenizer Helpers: uses · Tokenizer lessons rely on helper code
- Text Prep → Embeddings: feeds · Clean tokens feed embedding and position concepts
- Embeddings → Preprocess Pipeline: builds into · Representations are assembled into a full preprocessing pipeline
- Preprocess Pipeline → Attention: prepares for · Model-ready sequences feed transformer internals
- Attention → Transformer Blocks: composes · Attention is wrapped with transformer components
- Transformer Blocks → Mini Transformer: integrates · Components assemble into a minimal transformer
- Mini Transformer → Data & Tokenization: enables · A working transformer supports data and training work
- Data & Tokenization → Corpus Assets: reads · Tokenization and dataset notebooks consume shared corpora
- Data & Tokenization → Sequence Packing: feeds · Tokenized text is packed into training sequences
- Sequence Packing → Training Loop: streams into · Packed sequences enter the training loop
- Training Loop → Pretraining: drives · Training loop knowledge culminates in pretraining
- Pretraining → Alignment: base model for · Pretrained models become adaptation targets
- Alignment → Safety & Eval: evaluated by · Aligned models are checked for safety and quality
- Safety & Eval → Prompting & RAG: extends to · Post-alignment work shifts into prompting and retrieval
- Prompting & RAG → Retrieval: retrieves with · Prompting is augmented by retrieval components
- Retrieval → PEFT: adapts into · Retrieval-era models move to efficient specialization
- PEFT → Distillation: compresses into · Parameter-efficient methods lead to distillation and governance
- Distillation → Inference: deploys to · Specialized models are prepared for inference optimization
- Inference → Serving Stack: served by · Quantized and optimized models are exposed via serving stacks
- Serving Stack → Ops & Telemetry: operated with · Serving is managed through operational controls and telemetry
- Diagrams → Mini Transformer: illustrates · Reference visuals support conceptual understanding
- CI Workflow → LLM Curriculum: validates · Workflow supports reproducible notebook execution