# pobear1/nano-transformer architecture

> Architecture diagram of the GitHub repository pobear1/nano-transformer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pobear1/nano-transformer
- Repository: https://github.com/pobear1/nano-transformer

## Overview

1. **Project purpose**
   - This repo is a from-scratch reimplementation of a Transformer, built directly on PyTorch primitives rather than using PyTorch’s high-level attention modules.
   - The architecture is split between **core model building blocks** and **task-specific usage**, with the README indicating both training support and token-generation support.

2. **Core reusable primitives**
   - `basic_pieces.py` is the main repository-level implementation file for the foundational Transformer components.
   - This is likely where the low-level layers and math live: attention mechanics, feed-forward pieces, normalization, residual wiring, and other minimal building blocks needed to assemble the model.

3. **Chat-specific variant**
   - `chat/basic_pieces.py` suggests a second copy or adaptation of the same foundational pieces for the chat/token-generation path.
   - Architecturally, this implies a separation between the **generic Transformer core** and a **conversation/generation-oriented package** derived from Karpathy’s nanochat approach.

4. **Training and generation boundary**
   - The README makes clear that the repo includes not just model definition but also the **training framework required to get a working model**.
   - That means the architecture probably spans:
     - model construction,
     - training loop / optimization,
     - autoregressive token generation.
   - The generation path is conceptually downstream from the trained model weights and logits produced by the Transformer core.

5. **External conceptual sources**
   - The implementation is grounded in two architecture references:
     - *Attention Is All You Need* for the Transformer structure,
     - Murphy’s *Probabilistic Machine Learning* for the training/math framing.
   - For token generation, the repo borrows ideas from Andrej Karpathy’s `nanochat`, which likely affects sampling and decoding behavior.

6. **Vision data subsystem**
   - `vision/data/MNIST/...` contains raw MNIST files, so the repo also has a vision-related experiment area.
   - This is data-only in the tree provided, not a full vision model pipeline, so it should be treated as an **input dataset boundary** rather than a completed subsystem.

7. **Data flow in the vision area**
   - The MNIST raw gzip/ubyte files indicate the expected flow is:
     - raw dataset files on disk,
     - loader/preprocessing code elsewhere in the repo,
     - model training/evaluation on image data.
   - Since only the data files are present here, the architectural diagram should show MNIST as an external/local dataset source feeding a future or separate vision pipeline.

8. **Project dependencies and runtime**
   - `requirements.txt` indicates a Python/PyTorch-based runtime.
   - Given the README and file names, PyTorch is the central framework dependency, but the architecture intentionally avoids relying on built-in Transformer abstractions.

9. **Repository status and planning**
   - `progress.md` suggests active development tracking or implementation progress.
   - This is not part of the runtime architecture, but it indicates the codebase is likely still evolving and may have partially duplicated or experimental components.

10. **Important structural boundaries**
   - The main architectural boundary is between:
     - **framework-independent Transformer logic** in the basic pieces,
     - **application-specific logic** in chat and vision-related areas,
     - **local dataset assets** in `vision/data`.
   - For a diagram, keep the core model in the center, with task-specific paths branching out.

11. **What is notably absent**
   - There is no evidence in the provided tree of a full web service, API server, distributed training system, or model registry.
   - The repo appears to be a compact, research/education-oriented implementation rather than a production ML platform.

12. **Best way to draw it**
   - Show one central box for **Transformer primitives / training loop**.
   - Add a **chat generation path** fed by the same or a forked core implementation.
   - Add a separate **vision data input** box for MNIST, connected to a likely experimental vision pipeline.
   - Keep `requirements.txt` and `progress.md` as supporting project artifacts, not runtime services.

Main entry points: [`basic_pieces.py`](https://github.com/pobear1/nano-transformer/blob/main/basic_pieces.py) (Basic pieces), [`chat/basic_pieces.py`](https://github.com/pobear1/nano-transformer/blob/main/chat/basic_pieces.py) (Chat pieces), [`vision`](https://github.com/pobear1/nano-transformer/tree/main/vision) (Vision area), [`vision/data`](https://github.com/pobear1/nano-transformer/tree/main/vision/data) (Vision data)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Core transformer"]
  node_basic_pieces["Basic pieces<br/>transformer core<br/>[basic_pieces.py]"]
  node_training["Training loop<br/>train/runtime"]
  node_model_weights[("Model weights<br/>learned params")]
end

subgraph group_task["Task paths"]
  node_chat_pieces["Chat pieces<br/>generation variant<br/>[basic_pieces.py]"]
  node_generation["Token generation<br/>decode/runtime"]
  node_logits(("Logits<br/>model output"))
  node_chat_flow["Chat flow<br/>application path"]
  node_vision_area["Vision area<br/>experiment area"]
  node_vision_data[("Vision data<br/>data boundary")]
  node_mnist[("MNIST<br/>dataset")]
  node_mnist_raw[("Raw files<br/>dataset files")]
end

subgraph group_project["Project support"]
  node_readme["README<br/>project doc<br/>[README.md]"]
  node_requirements["Requirements<br/>python deps<br/>[requirements.txt]"]
  node_progress["Progress<br/>status note<br/>[progress.md]"]
  node_repo_root["Repo root<br/>project root"]
end

node_repo_root -->|"described by"| node_readme
node_repo_root -->|"depends on"| node_requirements
node_repo_root -.->|"tracked in"| node_progress
node_repo_root -->|"centers on"| node_basic_pieces
node_basic_pieces -->|"drives"| node_training
node_training -->|"produces"| node_model_weights
node_basic_pieces -->|"emits"| node_logits
node_logits -->|"consumed by"| node_generation
node_model_weights -->|"loaded by"| node_generation
node_basic_pieces -->|"forked into"| node_chat_pieces
node_chat_pieces -->|"supports"| node_chat_flow
node_chat_flow -->|"uses"| node_generation
node_vision_area -->|"contains"| node_vision_data
node_vision_data -->|"hosts"| node_mnist
node_mnist -->|"materialized as"| node_mnist_raw
node_mnist_raw -.->|"feeds"| node_vision_area
node_requirements -.->|"enables"| node_basic_pieces

click node_basic_pieces "https://github.com/pobear1/nano-transformer/blob/main/basic_pieces.py"
click node_chat_pieces "https://github.com/pobear1/nano-transformer/blob/main/chat/basic_pieces.py"
click node_vision_area "https://github.com/pobear1/nano-transformer/tree/main/vision"
click node_vision_data "https://github.com/pobear1/nano-transformer/tree/main/vision/data"
click node_mnist "https://github.com/pobear1/nano-transformer/tree/main/vision/data/MNIST"
click node_mnist_raw "https://github.com/pobear1/nano-transformer/tree/main/vision/data/MNIST/raw"
click node_readme "https://github.com/pobear1/nano-transformer/blob/main/README.md"
click node_requirements "https://github.com/pobear1/nano-transformer/blob/main/requirements.txt"
click node_progress "https://github.com/pobear1/nano-transformer/blob/main/progress.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_basic_pieces,node_training,node_model_weights toneBlue
class node_chat_pieces,node_generation,node_logits,node_chat_flow,node_vision_area,node_vision_data,node_mnist,node_mnist_raw toneAmber
class node_readme,node_requirements,node_progress,node_repo_root toneMint
```

## Components

### Core transformer

Reusable Transformer primitives and the main implementation boundary.

- [Basic pieces](https://github.com/pobear1/nano-transformer/blob/main/basic_pieces.py): transformer core · Primary PyTorch-based Transformer building blocks and low-level model logic. · `basic_pieces.py`
- **Training loop**: train/runtime · The training framework that turns Transformer components into a usable model.
- **Model weights**: learned params · Trained parameters produced by the training process and consumed by generation.

### Task paths

Specialized usage paths for chat generation and vision experiments.

- [Chat pieces](https://github.com/pobear1/nano-transformer/blob/main/chat/basic_pieces.py): generation variant · Chat-oriented adaptation of the core Transformer pieces for token generation. · `chat/basic_pieces.py`
- **Token generation**: decode/runtime · Autoregressive sampling and decoding path used after model training.
- **Logits**: model output · Next-token scores emitted by the Transformer core before sampling.
- **Chat flow**: application path · End-to-end chat-oriented usage built on the generation variant.
- [Vision area](https://github.com/pobear1/nano-transformer/tree/main/vision): experiment area · Separate experimental area for image data and future vision processing. · `vision`
- [Vision data](https://github.com/pobear1/nano-transformer/tree/main/vision/data): data boundary · Intermediate container for the MNIST dataset within the vision area. · `vision/data`
- [MNIST](https://github.com/pobear1/nano-transformer/tree/main/vision/data/MNIST): dataset · Raw MNIST image and label files used as local vision input. · `vision/data/MNIST`
- [Raw files](https://github.com/pobear1/nano-transformer/tree/main/vision/data/MNIST/raw): dataset files · Underlying gzip and IDX files for MNIST training and evaluation. · `vision/data/MNIST/raw`

### Project support

Documentation, dependencies, and development tracking.

- [README](https://github.com/pobear1/nano-transformer/blob/main/README.md): project doc · Project overview describing the Transformer implementation and usage. · `README.md`
- [Requirements](https://github.com/pobear1/nano-transformer/blob/main/requirements.txt): python deps · Python and PyTorch dependency specification for the repository. · `requirements.txt`
- [Progress](https://github.com/pobear1/nano-transformer/blob/main/progress.md): status note · Development progress tracking and implementation notes. · `progress.md`
- **Repo root**: project root · Top-level repository containing the core implementation and task-specific branches.

## Connections

- Repo root → README: described by · README explains the repository purpose and usage.
- Repo root → Requirements: depends on · Defines the Python runtime and library dependencies.
- Repo root → Progress: tracked in · Implementation progress and planning are recorded here.
- Repo root → Basic pieces: centers on · The repository's main implementation is built around the core Transformer pieces.
- Basic pieces → Training loop: drives · Core model components are assembled and trained through the training framework.
- Training loop → Model weights: produces · Training yields learned parameters for later use.
- Basic pieces → Logits: emits · The Transformer core produces next-token scores.
- Logits → Token generation: consumed by · Sampling and decoding operate on model logits.
- Model weights → Token generation: loaded by · Generation uses trained model parameters.
- Basic pieces → Chat pieces: forked into · Chat-specific code adapts the foundational implementation.
- Chat pieces → Chat flow: supports · The chat variant underpins the conversation-oriented path.
- Chat flow → Token generation: uses · Chat behavior depends on autoregressive generation.
- Vision area → Vision data: contains · The vision subtree contains the data boundary.
- Vision data → MNIST: hosts · MNIST is organized under the vision data subtree.
- MNIST → Raw files: materialized as · Raw IDX and gzip files form the dataset payload.
- Raw files → Vision area: feeds · Local image data is the input source for future vision experimentation.
- Requirements → Basic pieces: enables · PyTorch dependencies support the core implementation.
