# pritamn11/build-llm architecture

> Architecture diagram of the GitHub repository pritamn11/build-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pritamn11/build-llm
- Repository: https://github.com/pritamn11/build-llm

## Overview

1. **Repository shape**
   - This is a small, notebook-centric project: the core implementation lives in `Build_LLM.ipynb`, with a single training corpus in `a-man-thinketh.txt`, plus `requirements.txt` for Python dependencies.
   - There are no separate source packages, services, or deployment components; the architecture is a self-contained research/training workflow.

2. **Primary purpose**
   - The repo builds a **GPT-style causal language model from scratch** in PyTorch.
   - It is explicitly educational: the code is meant to show all major transformer pieces rather than reuse a pretrained model or a high-level framework wrapper.

3. **Data source boundary**
   - The only domain data is the text of **“As a Man Thinketh”** by James Allen.
   - That file is the model’s training corpus, so the entire learned distribution is anchored to a single small book rather than a large multi-document dataset.

4. **Text preprocessing layer**
   - The notebook likely handles raw-text ingestion, cleaning, and character/token-level preparation before training.
   - The readme indicates a GPT-2-sized vocabulary target (`50,257`), implying the preprocessing/tokenization path is aligned with GPT-2-style token IDs rather than a custom tiny vocabulary.

5. **Tokenization and sequence construction**
   - Input text is converted into fixed-length token sequences of **256 tokens**.
   - These sequences define the causal training examples: each position predicts the next token, preserving left-to-right autoregressive structure.

6. **Model core**
   - The model is a classic decoder-only transformer with:
     - token/position embeddings
     - **6 transformer layers**
     - **8 attention heads**
     - **256-dimensional embeddings**
   - This is the main computational subsystem and the central boundary between data preparation and learning.

7. **Attention and masking**
   - The architecture uses **multi-head self-attention** with **causal masking**, which prevents tokens from attending to future positions.
   - This masking is the key mechanism that makes the model suitable for next-token prediction and text generation.

8. **Feed-forward and normalization path**
   - Each transformer block includes a feed-forward network with **GELU** activation, **pre-norm layer normalization**, and **residual connections**.
   - These components stabilize optimization and preserve information across deep layers.

9. **Training loop**
   - The notebook contains the end-to-end training procedure: batch sampling, forward pass, loss computation, backward pass, optimizer step, and **gradient clipping**.
   - The architecture is likely single-process, notebook-executed training rather than distributed or service-based training.

10. **Generation/inference path**
   - After training, the model generates text autoregressively using **top-k sampling** and **temperature** control.
   - This is a separate inference flow from training: it starts from a prompt, repeatedly samples the next token, and feeds it back into the model.

11. **Runtime and dependencies**
   - The project is Python-based and centered on **PyTorch**.
   - `requirements.txt` likely captures the minimal ML stack needed to run the notebook, emphasizing local reproducibility over packaged deployment.

12. **Artifacts and outputs**
   - The main outputs are trained model weights, training metrics/loss curves, and generated sample text produced from the book-trained model.
   - Since there is no dedicated export or serving layer in the repo, the notebook is both the build environment and the execution environment.

13. **Important architectural boundary**
   - The key boundary in this repo is between:
     - **static training data**: one book in `a-man-thinketh.txt`
     - **model logic**: implemented in the notebook
   - There is no external API, database, or application backend; all behavior is local and model-centric.

14. **What to diagram**
   - For an architecture diagram, the main boxes should be:
     - text corpus
     - notebook-based preprocessing/tokenization
     - GPT-style transformer model
     - training loop with optimizer/clipping
     - autoregressive generation
   - The flow is one-way from corpus to tokens to model training, then from prompt to sampled text at inference time.

Main entry points: [`Build_LLM.ipynb`](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb) (Transformer stack), [`a-man-thinketh.txt`](https://github.com/pritamn11/build-llm/blob/main/a-man-thinketh.txt) (Book text), [`requirements.txt`](https://github.com/pritamn11/build-llm/blob/main/requirements.txt) (PyTorch stack)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data"]
  node_corpus["Book text<br/>training corpus<br/>[a-man-thinketh.txt]"]
end

subgraph group_notebook["Notebook Workflow"]
  node_notebook["Build notebook<br/>research notebook<br/>[Build_LLM.ipynb]"]
  node_deps["PyTorch stack<br/>python deps<br/>[requirements.txt]"]
  node_ingest["Text ingest<br/>raw-text load<br/>[Build_LLM.ipynb]"]
  node_clean["Preprocess text<br/>cleanup/token prep<br/>[Build_LLM.ipynb]"]
  node_tokenize["GPT-2 tokens<br/>tokenization<br/>[Build_LLM.ipynb]"]
  node_windows["256-token windows<br/>sequence builder<br/>[Build_LLM.ipynb]"]
end

subgraph group_model["Transformer Model"]
  node_embed["Embeddings<br/>token+position emb<br/>[Build_LLM.ipynb]"]
  node_blocks["Transformer stack<br/>6-layer decoder<br/>[Build_LLM.ipynb]"]
  node_attn["Causal attention<br/>multi-head self-attn<br/>[Build_LLM.ipynb]"]
  node_ffn["MLP path<br/>gelu feed-forward<br/>[Build_LLM.ipynb]"]
  node_norm["Pre-norm + skip<br/>layer norm/residual<br/>[Build_LLM.ipynb]"]
end

subgraph group_runtime["Runtime"]
  node_train["Training loop<br/>optimization loop<br/>[Build_LLM.ipynb]"]
  node_loss["Next-token loss<br/>causal objective<br/>[Build_LLM.ipynb]"]
  node_sample["Text sampler<br/>autoregressive inference<br/>[Build_LLM.ipynb]"]
  node_topk["Top-k sampling<br/>decode strategy<br/>[Build_LLM.ipynb]"]
  node_temp["Temperature<br/>decode control<br/>[Build_LLM.ipynb]"]
end

node_corpus -->|"source text"| node_ingest
node_ingest -->|"prepare"| node_clean
node_clean -->|"encode"| node_tokenize
node_tokenize -->|"slice"| node_windows
node_windows -->|"batches"| node_train
node_deps -.->|"runtime"| node_notebook
node_notebook -->|"implements"| node_ingest
node_notebook -->|"implements"| node_tokenize
node_notebook -->|"defines"| node_embed
node_embed -->|"feeds"| node_blocks
node_blocks -->|"contains"| node_attn
node_blocks -->|"contains"| node_ffn
node_blocks -->|"contains"| node_norm
node_attn -->|"stabilizes"| node_norm
node_ffn -->|"stabilizes"| node_norm
node_blocks -->|"predicts"| node_loss
node_train -->|"optimizes"| node_loss
node_train -->|"updates"| node_blocks
node_sample -->|"uses"| node_topk
node_sample -->|"uses"| node_temp
node_sample -->|"queries"| node_blocks
node_train -.->|"produces"| node_sample

click node_corpus "https://github.com/pritamn11/build-llm/blob/main/a-man-thinketh.txt"
click node_notebook "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_deps "https://github.com/pritamn11/build-llm/blob/main/requirements.txt"
click node_ingest "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_clean "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_tokenize "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_windows "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_embed "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_blocks "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_attn "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_ffn "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_norm "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_train "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_loss "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_sample "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_topk "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"
click node_temp "https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_corpus toneBlue
class node_notebook,node_deps,node_ingest,node_clean,node_tokenize,node_windows toneAmber
class node_embed,node_blocks,node_attn,node_ffn,node_norm toneMint
class node_train,node_loss,node_sample,node_topk,node_temp toneRose
```

## Components

### Data

Static training corpus and text source

- [Book text](https://github.com/pritamn11/build-llm/blob/main/a-man-thinketh.txt): training corpus · Single-book source text used as the only domain dataset · `a-man-thinketh.txt`

### Notebook Workflow

End-to-end implementation and execution in the notebook

- [Build notebook](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): research notebook · Main implementation for preprocessing, model definition, training, and sampling · `Build_LLM.ipynb`
- [PyTorch stack](https://github.com/pritamn11/build-llm/blob/main/requirements.txt): python deps · Minimal dependency specification for running the notebook · `requirements.txt`
- [Text ingest](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): raw-text load · Loads the source book and prepares it for processing · `Build_LLM.ipynb`
- [Preprocess text](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): cleanup/token prep · Cleans and prepares raw text for tokenization · `Build_LLM.ipynb`
- [GPT-2 tokens](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): tokenization · Converts text into GPT-2-style token ids and sequences · `Build_LLM.ipynb`
- [256-token windows](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): sequence builder · Builds fixed-length autoregressive training examples · `Build_LLM.ipynb`

### Transformer Model

Decoder-only GPT-style network internals

- [Embeddings](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): token+position emb · Maps token ids and positions into 256-dimensional vectors · `Build_LLM.ipynb`
- [Transformer stack](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): 6-layer decoder · Six causal transformer blocks in the GPT-style core · `Build_LLM.ipynb`
- [Causal attention](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): multi-head self-attn · Eight-head masked self-attention for next-token context · `Build_LLM.ipynb`
- [MLP path](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): gelu feed-forward · Per-block feed-forward network with GELU activation · `Build_LLM.ipynb`
- [Pre-norm + skip](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): layer norm/residual · Layer normalization and residual connections that stabilize depth · `Build_LLM.ipynb`

### Runtime

Training and generation execution paths

- [Training loop](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): optimization loop · Batching, forward pass, loss, backward pass, optimizer step, and gradient clipping · `Build_LLM.ipynb`
- [Next-token loss](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): causal objective · Computes autoregressive prediction loss over the token windows · `Build_LLM.ipynb`
- [Text sampler](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): autoregressive inference · Generates text from a prompt using iterative decoding · `Build_LLM.ipynb`
- [Top-k sampling](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): decode strategy · Restricts sampling to the most likely next-token candidates · `Build_LLM.ipynb`
- [Temperature](https://github.com/pritamn11/build-llm/blob/main/Build_LLM.ipynb): decode control · Adjusts randomness during generation · `Build_LLM.ipynb`

## Connections

- Book text → Text ingest: source text · The book text is loaded into the notebook for processing
- Text ingest → Preprocess text: prepare · Raw text is cleaned before tokenization
- Preprocess text → GPT-2 tokens: encode · Prepared text is converted into token ids
- GPT-2 tokens → 256-token windows: slice · Token stream is cut into fixed-length causal sequences
- 256-token windows → Training loop: batches · Training examples are sampled from token windows
- PyTorch stack → Build notebook: runtime · Dependencies support the notebook execution environment
- Build notebook → Text ingest: implements · Notebook contains the data ingestion step
- Build notebook → GPT-2 tokens: implements · Notebook defines the tokenization and sequence construction path
- Build notebook → Embeddings: defines · Notebook implements the model embedding layers
- Embeddings → Transformer stack: feeds · Embedded tokens enter the transformer stack
- Transformer stack → Causal attention: contains · Each transformer block uses masked multi-head attention
- Transformer stack → MLP path: contains · Each transformer block includes a feed-forward MLP
- Transformer stack → Pre-norm + skip: contains · Each block uses pre-norm and residual connections
- Causal attention → Pre-norm + skip: stabilizes · Attention output is normalized and combined with residual paths
- MLP path → Pre-norm + skip: stabilizes · MLP output is normalized and combined with residual paths
- Transformer stack → Next-token loss: predicts · Transformer outputs are used to compute next-token loss
- Training loop → Next-token loss: optimizes · The training loop minimizes causal prediction loss
- Training loop → Transformer stack: updates · Backpropagation updates the transformer parameters
- Text sampler → Top-k sampling: uses · Generation applies top-k filtering during decoding
- Text sampler → Temperature: uses · Generation applies temperature control during decoding
- Text sampler → Transformer stack: queries · Autoregressive decoding repeatedly runs the trained transformer
- Training loop → Text sampler: produces · Training yields a model used for text generation