# avalokkumar/learning-llm architecture

> Architecture diagram of the GitHub repository avalokkumar/learning-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/avalokkumar/learning-llm
- Repository: https://github.com/avalokkumar/learning-llm

## Overview

1. This repository is an educational LLM curriculum, not a single application. Its architecture is organized as a linear learning pipeline: each week builds on the previous one, moving from text preprocessing to transformer internals, training, alignment, retrieval, fine-tuning, and inference optimization.

2. The primary unit of composition is the Jupyter notebook. Each `weekX/dayY` folder is a self-contained lesson with one or more notebooks and a short `README.md` that frames the topic. For an architecture diagram, these notebooks should be treated as modular learning/workflow nodes rather than runtime services.

3. Week 1 is the text-processing foundation. It covers normalization, tokenization, embeddings, positional encodings, and a full preprocessing pipeline. The only visible code-first helpers live here: `week1/day1/bert_tokenizer.py` and `tokenizer_examples.py`, which support the tokenization lessons and reinforce the boundary between raw text and model-ready tokens.

4. Week 2 is the transformer core. The notebooks progress through scaled dot-product attention, multi-head attention, transformer variants, component layers such as residuals and layer norm, and then a minimal end-to-end transformer. This is the main model-architecture subsystem in the repo.

5. Week 3 focuses on training data and pretraining. It includes dataset/perplexity work, tokenization at scale, sequence packing, the training loop, regularization/monitoring, and a nanoGPT-style pretraining notebook. This week is the bridge from “model components” to “trainable language model,” with practical dataflow from text corpora into tokenized training sequences.

6. Week 3 also contains concrete artifacts that act as inputs to later notebooks: `week3/day12/data/*.txt` provides sample corpora, and `week3/day12/tokenizers/*.json` stores trained tokenizer vocabularies for BPE, Unigram, and WordPiece. These are the main persistent data assets in the repo.

7. Week 4 is the alignment and evaluation layer. It covers supervised fine-tuning, RLHF components, DPO, safety implementation, and evaluation. Architecturally, this is the post-pretraining adaptation boundary: base model outputs are refined with instruction data, preference data, safety rules, and benchmark-driven evaluation.

8. Week 5 shifts to prompting and retrieval-augmented generation. The notebooks move from prompt engineering to RAG, advanced retrieval, RAG optimization, and RAG evaluation/safety. This introduces an external-knowledge path: queries are augmented with retrieved context before generation, rather than relying only on the parametric model.

9. Week 6 is parameter-efficient adaptation and compression-oriented training. It includes LoRA, QLoRA, distillation, catastrophic forgetting, hyperparameter tuning, early stopping, adapter merging, and model cards. This subsystem emphasizes efficient model specialization and model governance artifacts.

10. Week 7 covers inference-time optimization and serving. The notebooks address quantization, KV caching, flash attention, paged attention, batching strategies, scheduling, local model serving, streaming APIs, TGI, TensorRT-LLM, autoscaling, caching, rate limiting, multi-tenancy, metrics, logging, tracing, GPU monitoring, and telemetry. For a diagram, this is the deployment/runtime boundary where trained models become serviced workloads.

11. The `diagrams/` folder is the visual support layer for the curriculum. It is not the core learning path, but it likely contains reference graphics used to explain model internals and system relationships.

12. `.github/workflows/python-package-conda.yml` is the only explicit automation boundary visible. It suggests basic CI/environment validation around the Python/conda setup, supporting reproducibility for notebook execution rather than deploying an application.

13. Overall data flow in the repo is instructional: raw text and corpora feed tokenizers; tokenized sequences feed embeddings and transformers; trained base models feed fine-tuning, preference optimization, and safety work; then retrieval, quantization, batching, and serving turn the models into usable systems. The architecture is best drawn as a staged curriculum with shared assets and artifacts crossing week boundaries.

Main entry points: [`week1/day1/day1_tokenization_practical.ipynb`](https://github.com/avalokkumar/learning-llm/blob/main/week1/day1/day1_tokenization_practical.ipynb) (Text Prep), [`week3/day12/day12_tokenization_part1.ipynb`](https://github.com/avalokkumar/learning-llm/blob/main/week3/day12/day12_tokenization_part1.ipynb) (Data & Tokenization), [`week2/day10/day10_minimal_transformer_part1.ipynb`](https://github.com/avalokkumar/learning-llm/blob/main/week2/day10/day10_minimal_transformer_part1.ipynb) (Mini Transformer), [`README.md`](https://github.com/avalokkumar/learning-llm/blob/main/README.md) (LLM Curriculum)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_curriculum["Curriculum"]
  node_repo_root["LLM Curriculum<br/>curriculum-root<br/>[README.md]"]
  node_w1_text["Text Prep<br/>week1-foundation"]
  node_w1_repr["Embeddings<br/>week1-representations"]
  node_w1_pipeline["Preprocess Pipeline<br/>week1-pipeline"]
  node_w2_attention["Attention<br/>week2-core"]
  node_w2_blocks["Transformer Blocks<br/>week2-components"]
  node_w2_model["Mini Transformer<br/>week2-model"]
  node_w3_data["Data & Tokenization<br/>week3-data"]
  node_w3_packing["Sequence Packing<br/>week3-packaging"]
  node_w3_train["Training Loop<br/>week3-training"]
  node_w3_pretrain["Pretraining"]
  node_w4_align["Alignment<br/>week4-adaptation"]
  node_w4_safety["Safety & Eval<br/>week4-evaluation"]
  node_w5_prompt_rag["Prompting & RAG<br/>week5-retrieval"]
  node_w5_retrieval["Retrieval"]
  node_w6_peft["PEFT<br/>week6-efficient-adaptation"]
  node_w6_distill["Distillation<br/>week6-compression"]
  node_w7_infer["Inference<br/>week7-runtime"]
  node_w7_serving["Serving Stack<br/>week7-serving"]
  node_w7_ops["Ops & Telemetry<br/>week7-operations"]
end

subgraph group_assets["Shared Assets"]
  node_w1_helpers["Tokenizer Helpers<br/>week1-code<br/>[bert_tokenizer.py]"]
  node_w3_assets[("Corpus Assets<br/>dataset-assets<br/>[all_texts.txt]")]
  node_diagrams["Diagrams<br/>visual-support"]
end

subgraph group_support["Support"]
  node_ci["CI Workflow<br/>automation"]
end

node_repo_root -->|"starts with"| node_w1_text
node_w1_text -->|"uses"| node_w1_helpers
node_w1_text -->|"feeds"| node_w1_repr
node_w1_repr -->|"builds into"| node_w1_pipeline
node_w1_pipeline -->|"prepares for"| node_w2_attention
node_w2_attention -->|"composes"| node_w2_blocks
node_w2_blocks -->|"integrates"| node_w2_model
node_w2_model -->|"enables"| node_w3_data
node_w3_data -->|"reads"| node_w3_assets
node_w3_data -->|"feeds"| node_w3_packing
node_w3_packing -->|"streams into"| node_w3_train
node_w3_train -->|"drives"| node_w3_pretrain
node_w3_pretrain -->|"base model for"| node_w4_align
node_w4_align -->|"evaluated by"| node_w4_safety
node_w4_safety -->|"extends to"| node_w5_prompt_rag
node_w5_prompt_rag -->|"retrieves with"| node_w5_retrieval
node_w5_retrieval -->|"adapts into"| node_w6_peft
node_w6_peft -->|"compresses into"| node_w6_distill
node_w6_distill -->|"deploys to"| node_w7_infer
node_w7_infer -->|"served by"| node_w7_serving
node_w7_serving -->|"operated with"| node_w7_ops
node_diagrams -.->|"illustrates"| node_w2_model
node_ci -.->|"validates"| node_repo_root

click node_repo_root "https://github.com/avalokkumar/learning-llm/blob/main/README.md"
click node_w1_text "https://github.com/avalokkumar/learning-llm/blob/main/week1/day1/day1_tokenization_practical.ipynb"
click node_w1_helpers "https://github.com/avalokkumar/learning-llm/blob/main/week1/day1/bert_tokenizer.py"
click node_w1_repr "https://github.com/avalokkumar/learning-llm/blob/main/week1/day3/day3_embeddings_part1.ipynb"
click node_w1_pipeline "https://github.com/avalokkumar/learning-llm/blob/main/week1/day5/day5_pipeline_part1.ipynb"
click node_w2_attention "https://github.com/avalokkumar/learning-llm/blob/main/week2/day6/day6_attention_implementation.ipynb"
click node_w2_blocks "https://github.com/avalokkumar/learning-llm/blob/main/week2/day9/day9_transformer_components_part1.ipynb"
click node_w2_model "https://github.com/avalokkumar/learning-llm/blob/main/week2/day10/day10_minimal_transformer_part1.ipynb"
click node_w3_data "https://github.com/avalokkumar/learning-llm/blob/main/week3/day12/day12_tokenization_part1.ipynb"
click node_w3_assets "https://github.com/avalokkumar/learning-llm/blob/main/week3/day12/data/all_texts.txt"
click node_w3_packing "https://github.com/avalokkumar/learning-llm/blob/main/week3/day12/day12_sequence_packing.ipynb"
click node_w3_train "https://github.com/avalokkumar/learning-llm/blob/main/week3/day13/day13_training_loop_complete.ipynb"
click node_w3_pretrain "https://github.com/avalokkumar/learning-llm/blob/main/week3/day15/day15_nanogpt_pretraining.ipynb"
click node_w4_align "https://github.com/avalokkumar/learning-llm/blob/main/week4/day16/day16_supervised_fine_tuning.ipynb"
click node_w4_safety "https://github.com/avalokkumar/learning-llm/blob/main/week4/day19/day19_safety_implementation.ipynb"
click node_w5_prompt_rag "https://github.com/avalokkumar/learning-llm/blob/main/week5/day22/day22_rag_implementation.ipynb"
click node_w5_retrieval "https://github.com/avalokkumar/learning-llm/blob/main/week5/day23/day23_advanced_retrieval.ipynb"
click node_w6_peft "https://github.com/avalokkumar/learning-llm/blob/main/week6/day26/day26_lora_implementation_part1.ipynb"
click node_w6_distill "https://github.com/avalokkumar/learning-llm/blob/main/week6/day28/day28_distillation_implementation_part1.ipynb"
click node_w7_infer "https://github.com/avalokkumar/learning-llm/blob/main/week7/day31/day31_int8_quantization_part1.ipynb"
click node_w7_serving "https://github.com/avalokkumar/learning-llm/blob/main/week7/day33/day33_local_model_server_part1.ipynb"
click node_w7_ops "https://github.com/avalokkumar/learning-llm/blob/main/week7/day34/day34_caching_strategies_part1.ipynb"
click node_diagrams "https://github.com/avalokkumar/learning-llm/tree/main/diagrams"
click node_ci "https://github.com/avalokkumar/learning-llm/blob/main/.github/workflows/python-package-conda.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_repo_root,node_w1_text,node_w1_repr,node_w1_pipeline,node_w2_attention,node_w2_blocks,node_w2_model,node_w3_data,node_w3_packing,node_w3_train,node_w3_pretrain,node_w4_align,node_w4_safety,node_w5_prompt_rag,node_w5_retrieval,node_w6_peft,node_w6_distill,node_w7_infer,node_w7_serving,node_w7_ops toneBlue
class node_w1_helpers,node_w3_assets,node_diagrams toneAmber
class node_ci toneMint
```

## Components

### Curriculum

Main week-by-week learning pipeline

- [LLM Curriculum](https://github.com/avalokkumar/learning-llm/blob/main/README.md): curriculum-root · Top-level educational path across all weeks · `README.md`
- [Text Prep](https://github.com/avalokkumar/learning-llm/blob/main/week1/day1/day1_tokenization_practical.ipynb): week1-foundation · Normalization and tokenization lessons · `week1/day1/day1_tokenization_practical.ipynb`
- [Embeddings](https://github.com/avalokkumar/learning-llm/blob/main/week1/day3/day3_embeddings_part1.ipynb): week1-representations · Embeddings and positional encodings · `week1/day3/day3_embeddings_part1.ipynb`
- [Preprocess Pipeline](https://github.com/avalokkumar/learning-llm/blob/main/week1/day5/day5_pipeline_part1.ipynb): week1-pipeline · End-to-end text-to-model-ready pipeline · `week1/day5/day5_pipeline_part1.ipynb`
- [Attention](https://github.com/avalokkumar/learning-llm/blob/main/week2/day6/day6_attention_implementation.ipynb): week2-core · Scaled dot-product and multi-head attention · `week2/day6/day6_attention_implementation.ipynb`
- [Transformer Blocks](https://github.com/avalokkumar/learning-llm/blob/main/week2/day9/day9_transformer_components_part1.ipynb): week2-components · Residuals, layer norm, and transformer variants · `week2/day9/day9_transformer_components_part1.ipynb`
- [Mini Transformer](https://github.com/avalokkumar/learning-llm/blob/main/week2/day10/day10_minimal_transformer_part1.ipynb): week2-model · Minimal end-to-end transformer model · `week2/day10/day10_minimal_transformer_part1.ipynb`
- [Data & Tokenization](https://github.com/avalokkumar/learning-llm/blob/main/week3/day12/day12_tokenization_part1.ipynb): week3-data · Datasets, corpora, and tokenizer assets · `week3/day12/day12_tokenization_part1.ipynb`
- [Sequence Packing](https://github.com/avalokkumar/learning-llm/blob/main/week3/day12/day12_sequence_packing.ipynb): week3-packaging · Packing tokenized examples for training · `week3/day12/day12_sequence_packing.ipynb`
- [Training Loop](https://github.com/avalokkumar/learning-llm/blob/main/week3/day13/day13_training_loop_complete.ipynb): week3-training · Core training loop and optimization flow · `week3/day13/day13_training_loop_complete.ipynb`
- [Pretraining](https://github.com/avalokkumar/learning-llm/blob/main/week3/day15/day15_nanogpt_pretraining.ipynb): week3-pretraining · NanoGPT-style pretraining notebook · `week3/day15/day15_nanogpt_pretraining.ipynb`
- [Alignment](https://github.com/avalokkumar/learning-llm/blob/main/week4/day16/day16_supervised_fine_tuning.ipynb): week4-adaptation · Supervised fine-tuning, RLHF, and DPO · `week4/day16/day16_supervised_fine_tuning.ipynb`
- [Safety & Eval](https://github.com/avalokkumar/learning-llm/blob/main/week4/day19/day19_safety_implementation.ipynb): week4-evaluation · Safety implementation and benchmark evaluation · `week4/day19/day19_safety_implementation.ipynb`
- [Prompting & RAG](https://github.com/avalokkumar/learning-llm/blob/main/week5/day22/day22_rag_implementation.ipynb): week5-retrieval · Prompt engineering and retrieval-augmented generation · `week5/day22/day22_rag_implementation.ipynb`
- [Retrieval](https://github.com/avalokkumar/learning-llm/blob/main/week5/day23/day23_advanced_retrieval.ipynb): week5-retrieval-core · Advanced retrieval and RAG optimization · `week5/day23/day23_advanced_retrieval.ipynb`
- [PEFT](https://github.com/avalokkumar/learning-llm/blob/main/week6/day26/day26_lora_implementation_part1.ipynb): week6-efficient-adaptation · LoRA, QLoRA, and adapter-based specialization · `week6/day26/day26_lora_implementation_part1.ipynb`
- [Distillation](https://github.com/avalokkumar/learning-llm/blob/main/week6/day28/day28_distillation_implementation_part1.ipynb): week6-compression · Distillation, forgetting, tuning, and model governance · `week6/day28/day28_distillation_implementation_part1.ipynb`
- [Inference](https://github.com/avalokkumar/learning-llm/blob/main/week7/day31/day31_int8_quantization_part1.ipynb): week7-runtime · Quantization and decoding-time acceleration · `week7/day31/day31_int8_quantization_part1.ipynb`
- [Serving Stack](https://github.com/avalokkumar/learning-llm/blob/main/week7/day33/day33_local_model_server_part1.ipynb): week7-serving · Local serving, APIs, batching, and deployment · `week7/day33/day33_local_model_server_part1.ipynb`
- [Ops & Telemetry](https://github.com/avalokkumar/learning-llm/blob/main/week7/day34/day34_caching_strategies_part1.ipynb): week7-operations · Caching, rate limiting, multitenancy, logging, and monitoring · `week7/day34/day34_caching_strategies_part1.ipynb`

### Shared Assets

Persistent data, helper code, and visual references

- [Tokenizer Helpers](https://github.com/avalokkumar/learning-llm/blob/main/week1/day1/bert_tokenizer.py): week1-code · Code-first tokenizer boundary for lessons · `week1/day1/bert_tokenizer.py`
- [Corpus Assets](https://github.com/avalokkumar/learning-llm/blob/main/week3/day12/data/all_texts.txt): dataset-assets · Sample text corpora and trained tokenizer vocabularies · `week3/day12/data/all_texts.txt`
- [Diagrams](https://github.com/avalokkumar/learning-llm/tree/main/diagrams): visual-support · Reference graphics for the curriculum · `diagrams`

### Support

Repository automation and reproducibility

- [CI Workflow](https://github.com/avalokkumar/learning-llm/blob/main/.github/workflows/python-package-conda.yml): automation · Python/conda validation and environment setup · `.github/workflows/python-package-conda.yml`

## Connections

- LLM Curriculum → Text Prep: starts with · Curriculum begins with text preprocessing and tokenization
- Text Prep → Tokenizer Helpers: uses · Tokenizer lessons rely on helper code
- Text Prep → Embeddings: feeds · Clean tokens feed embedding and position concepts
- Embeddings → Preprocess Pipeline: builds into · Representations are assembled into a full preprocessing pipeline
- Preprocess Pipeline → Attention: prepares for · Model-ready sequences feed transformer internals
- Attention → Transformer Blocks: composes · Attention is wrapped with transformer components
- Transformer Blocks → Mini Transformer: integrates · Components assemble into a minimal transformer
- Mini Transformer → Data & Tokenization: enables · A working transformer supports data and training work
- Data & Tokenization → Corpus Assets: reads · Tokenization and dataset notebooks consume shared corpora
- Data & Tokenization → Sequence Packing: feeds · Tokenized text is packed into training sequences
- Sequence Packing → Training Loop: streams into · Packed sequences enter the training loop
- Training Loop → Pretraining: drives · Training loop knowledge culminates in pretraining
- Pretraining → Alignment: base model for · Pretrained models become adaptation targets
- Alignment → Safety & Eval: evaluated by · Aligned models are checked for safety and quality
- Safety & Eval → Prompting & RAG: extends to · Post-alignment work shifts into prompting and retrieval
- Prompting & RAG → Retrieval: retrieves with · Prompting is augmented by retrieval components
- Retrieval → PEFT: adapts into · Retrieval-era models move to efficient specialization
- PEFT → Distillation: compresses into · Parameter-efficient methods lead to distillation and governance
- Distillation → Inference: deploys to · Specialized models are prepared for inference optimization
- Inference → Serving Stack: served by · Quantized and optimized models are exposed via serving stacks
- Serving Stack → Ops & Telemetry: operated with · Serving is managed through operational controls and telemetry
- Diagrams → Mini Transformer: illustrates · Reference visuals support conceptual understanding
- CI Workflow → LLM Curriculum: validates · Workflow supports reproducible notebook execution
