# valkmjolnir/quetzalcoatlus architecture

> Architecture diagram of the GitHub repository valkmjolnir/quetzalcoatlus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/valkmjolnir/quetzalcoatlus
- Repository: https://github.com/valkmjolnir/quetzalcoatlus

## Overview

Quetzalcoatlus is an experimental GPT-2 training process and inference engine. Its documented workflow prepares text or conversation data, tokenizes it, trains a base model, fine-tunes it with SFT, and generates text from a checkpoint. A separate C++ inference engine loads exported weights and tokenizer data for chat and diagnostic modes. The graph preserves these distinct paths and the shared model, tokenizer, and tensor layers. Some training and inference modules were not sampled; their documented capabilities are included, but uncertain internal calls are omitted.

Main entry points: [`quetzal/lib/mode/chat.cpp`](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/chat.cpp) (Interactive chat), [`gpt/generate.py`](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/generate.py) (Python generation), [`quetzal/deploy.cpp`](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/deploy.cpp) (C++ mode dispatcher), [`gpt/tokenizer.py`](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/tokenizer.py) (Python tokenizer and binning)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_concat["JSONL concatenation<br/>[concat_data_set.py]"]
  node_pre_split["Pretrain splitting"]
  node_sft_split["SFT splitting"]
end

subgraph group_python["Python training and inference"]
  node_py_tokenizer["Python tokenizer and binning<br/>[tokenizer.py]"]
  node_pre_loader["Pretrain data loader<br/>[pre_dataloader.py]"]
  node_sft_loader["SFT data loader<br/>[sft_dataloader.py]"]
  node_pretrain["Pretraining<br/>[pre_training.py]"]
  node_sft_train["SFT training<br/>[sft_training.py]"]
  node_py_model["Python GPT model<br/>[gpt.py]"]
  node_py_transformer["Python transformer<br/>[transformer.py]"]
  node_py_attention["Python attention<br/>[attn.py]"]
  node_py_generate["Python generation<br/>[generate.py]"]
  node_device["Device selection<br/>[device.py]"]
end

subgraph group_cpp["C++ inference engine"]
  node_cpp_tokenizer_build["BBPE tokenizer builder<br/>[tokenizer.cpp]"]
  node_tokenizer_export["Tokenizer export<br/>[tokenizer.cpp]"]
  node_deploy["C++ mode dispatcher<br/>[deploy.cpp]"]
  node_chat["Interactive chat<br/>[chat.cpp]"]
  node_attn_mode["Attention inspection<br/>[attn.cpp]"]
  node_perf_mode["Layer performance<br/>[performance.cpp]"]
  node_experiment["Tensor experiments<br/>[experiment.cpp]"]
  node_bin_reader["Tokenizer binary reader<br/>[bin_reader.cpp]"]
  node_cpp_tokenizer["BBPE encode and decode<br/>[tokenizer.hpp]"]
end

subgraph group_core["Model and tensor core"]
  node_weights[("Weight loading")]
  node_cpp_model["C++ GPT-2 model<br/>[gpt2.cpp]"]
  node_cpp_transformer["C++ transformer block<br/>[transformer.hpp]"]
  node_cpp_attention["C++ attention<br/>[attention.cpp]"]
  node_tensor["Tensor operations<br/>[tensor.hpp]"]
  node_linalg["Linear algebra<br/>[linalg.hpp]"]
  node_rope["Rotary embeddings<br/>[rope.hpp]"]
end

node_dataset["Training dataset"]
node_cli_user(("CLI user"))
node_chat_user(("Chat user"))
node_generated["Generated text"]
node_diagnostics["Diagnostic output"]

node_dataset -->|"concatenates"| node_concat
node_concat -->|"prepares text"| node_pre_split
node_dataset -->|"splits conversations"| node_sft_split
node_pre_split -->|"supplies text"| node_py_tokenizer
node_py_tokenizer -->|"writes token bins"| node_pre_loader
node_pre_loader -.->|"provides batches"| node_pretrain
node_pretrain -.->|"trains"| node_py_model
node_sft_split -->|"supplies conversations"| node_sft_loader
node_sft_loader -->|"encodes text"| node_py_tokenizer
node_sft_loader -.->|"provides batches"| node_sft_train
node_sft_train -.->|"fine-tunes"| node_py_model
node_cli_user -->|"requests generation"| node_py_generate
node_py_generate -->|"encodes prompts"| node_py_tokenizer
node_py_generate -->|"runs inference"| node_py_model
node_py_generate -->|"selects device"| node_device
node_py_model -->|"uses blocks"| node_py_transformer
node_py_transformer -.->|"uses attention"| node_py_attention
node_py_generate -->|"returns text"| node_generated
node_cli_user -->|"builds vocabulary"| node_cpp_tokenizer_build
node_cpp_tokenizer_build -->|"exports tokenizer"| node_tokenizer_export
node_cli_user -->|"selects mode"| node_deploy
node_deploy -->|"dispatches chat"| node_chat
node_deploy -->|"dispatches attention"| node_attn_mode
node_deploy -->|"dispatches performance"| node_perf_mode
node_deploy -->|"dispatches experiment"| node_experiment
node_chat_user -->|"enters conversation"| node_chat
node_chat -->|"loads weights"| node_weights
node_chat -->|"loads tokenizer data"| node_bin_reader
node_chat -->|"encodes and decodes"| node_cpp_tokenizer
node_chat -->|"runs generation"| node_cpp_model
node_cpp_model -.->|"reads parameters"| node_weights
node_cpp_model -.->|"uses blocks"| node_cpp_transformer
node_cpp_transformer -->|"owns attention"| node_cpp_attention
node_cpp_attention -->|"applies rotations"| node_rope
node_cpp_attention -->|"uses operations"| node_linalg
node_cpp_transformer -->|"uses tensors"| node_tensor
node_chat -->|"streams reply"| node_generated
node_attn_mode -->|"shows attention"| node_diagnostics
node_perf_mode -->|"reports timings"| node_diagnostics
node_experiment -->|"writes tensor dumps"| node_diagnostics

click node_concat "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/tool/concat_data_set.py"
click node_pre_split "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/tool/split_pre_train_data.py"
click node_sft_split "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/tool/split_sft_train_data.py"
click node_cpp_tokenizer_build "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/bbpe/tokenizer.cpp"
click node_tokenizer_export "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/tokenizer.cpp"
click node_py_tokenizer "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/tokenizer.py"
click node_pre_loader "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/pre_dataloader.py"
click node_sft_loader "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/sft_dataloader.py"
click node_pretrain "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/pre_training.py"
click node_sft_train "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/sft_training.py"
click node_py_model "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/gpt.py"
click node_py_transformer "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/transformer.py"
click node_py_attention "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/attn.py"
click node_py_generate "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/generate.py"
click node_device "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/lib/device.py"
click node_deploy "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/deploy.cpp"
click node_chat "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/chat.cpp"
click node_attn_mode "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/attn.cpp"
click node_perf_mode "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/performance.cpp"
click node_experiment "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/experiment.cpp"
click node_weights "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/tensor/weights_manager.cpp"
click node_cpp_model "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/gpt/gpt2.cpp"
click node_cpp_transformer "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/gpt/transformer.hpp"
click node_cpp_attention "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/gpt/attention.cpp"
click node_tensor "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/tensor/tensor.hpp"
click node_linalg "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/tensor/linalg.hpp"
click node_rope "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/tensor/rope.hpp"
click node_bin_reader "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/bbpe/bin_reader.cpp"
click node_cpp_tokenizer "https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/bbpe/tokenizer.hpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_concat,node_pre_split,node_sft_split,node_cli_user,node_chat_user toneBlue
class node_py_tokenizer,node_pre_loader,node_sft_loader,node_pretrain,node_sft_train,node_py_model,node_py_transformer,node_py_attention,node_py_generate,node_device toneAmber
class node_cpp_tokenizer_build,node_tokenizer_export,node_deploy,node_chat,node_attn_mode,node_perf_mode,node_experiment,node_bin_reader,node_cpp_tokenizer toneMint
class node_weights,node_cpp_model,node_cpp_transformer,node_cpp_attention,node_tensor,node_linalg,node_rope toneRose
class node_dataset,node_generated,node_diagnostics toneIndigo
```

## Components

### Data preparation

- [JSONL concatenation](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/tool/concat_data_set.py): `tool/concat_data_set.py`
- [Pretrain splitting](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/tool/split_pre_train_data.py): `tool/split_pre_train_data.py`
- [SFT splitting](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/tool/split_sft_train_data.py): `tool/split_sft_train_data.py`

### Python training and inference

- [Python tokenizer and binning](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/tokenizer.py): `gpt/tokenizer.py`
- [Pretrain data loader](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/pre_dataloader.py): `gpt/pre_dataloader.py`
- [SFT data loader](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/sft_dataloader.py): `gpt/sft_dataloader.py`
- [Pretraining](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/pre_training.py): `gpt/pre_training.py`
- [SFT training](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/sft_training.py): `gpt/sft_training.py`
- [Python GPT model](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/gpt.py): `gpt/gpt.py`
- [Python transformer](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/transformer.py): `gpt/transformer.py`
- [Python attention](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/attn.py): `gpt/attn.py`
- [Python generation](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/generate.py): `gpt/generate.py`
- [Device selection](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/gpt/lib/device.py): `gpt/lib/device.py`

### C++ inference engine

- [BBPE tokenizer builder](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/bbpe/tokenizer.cpp): `quetzal/lib/bbpe/tokenizer.cpp`
- [Tokenizer export](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/tokenizer.cpp): `quetzal/tokenizer.cpp`
- [C++ mode dispatcher](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/deploy.cpp): `quetzal/deploy.cpp`
- [Interactive chat](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/chat.cpp): `quetzal/lib/mode/chat.cpp`
- [Attention inspection](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/attn.cpp): `quetzal/lib/mode/attn.cpp`
- [Layer performance](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/performance.cpp): `quetzal/lib/mode/performance.cpp`
- [Tensor experiments](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/mode/experiment.cpp): `quetzal/lib/mode/experiment.cpp`
- [Tokenizer binary reader](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/bbpe/bin_reader.cpp): `quetzal/lib/bbpe/bin_reader.cpp`
- [BBPE encode and decode](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/bbpe/tokenizer.hpp): `quetzal/include/bbpe/tokenizer.hpp`

### Model and tensor core

- [Weight loading](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/tensor/weights_manager.cpp): `quetzal/lib/tensor/weights_manager.cpp`
- [C++ GPT-2 model](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/gpt/gpt2.cpp): `quetzal/lib/gpt/gpt2.cpp`
- [C++ transformer block](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/gpt/transformer.hpp): `quetzal/include/gpt/transformer.hpp`
- [C++ attention](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/lib/gpt/attention.cpp): `quetzal/lib/gpt/attention.cpp`
- [Tensor operations](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/tensor/tensor.hpp): `quetzal/include/tensor/tensor.hpp`
- [Linear algebra](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/tensor/linalg.hpp): `quetzal/include/tensor/linalg.hpp`
- [Rotary embeddings](https://github.com/valkmjolnir/quetzalcoatlus/blob/main/quetzal/include/tensor/rope.hpp): `quetzal/include/tensor/rope.hpp`

### Other components

- **Training dataset**
- **CLI user**
- **Chat user**
- **Generated text**
- **Diagnostic output**

## Connections

- Training dataset → JSONL concatenation: concatenates
- JSONL concatenation → Pretrain splitting: prepares text
- Training dataset → SFT splitting: splits conversations
- Pretrain splitting → Python tokenizer and binning: supplies text
- Python tokenizer and binning → Pretrain data loader: writes token bins
- Pretrain data loader → Pretraining: provides batches
- Pretraining → Python GPT model: trains
- SFT splitting → SFT data loader: supplies conversations
- SFT data loader → Python tokenizer and binning: encodes text (evidence: `gpt/sft_dataloader.py`)
- SFT data loader → SFT training: provides batches
- SFT training → Python GPT model: fine-tunes
- CLI user → Python generation: requests generation
- Python generation → Python tokenizer and binning: encodes prompts (evidence: `gpt/generate.py`)
- Python generation → Python GPT model: runs inference (evidence: `gpt/generate.py`)
- Python generation → Device selection: selects device (evidence: `gpt/generate.py`)
- Python GPT model → Python transformer: uses blocks (evidence: `gpt/gpt.py`)
- Python transformer → Python attention: uses attention
- Python generation → Generated text: returns text
- CLI user → BBPE tokenizer builder: builds vocabulary
- BBPE tokenizer builder → Tokenizer export: exports tokenizer (evidence: `quetzal/tokenizer.cpp`)
- CLI user → C++ mode dispatcher: selects mode (evidence: `quetzal/deploy.cpp`)
- C++ mode dispatcher → Interactive chat: dispatches chat (evidence: `quetzal/deploy.cpp`)
- C++ mode dispatcher → Attention inspection: dispatches attention (evidence: `quetzal/deploy.cpp`)
- C++ mode dispatcher → Layer performance: dispatches performance (evidence: `quetzal/deploy.cpp`)
- C++ mode dispatcher → Tensor experiments: dispatches experiment (evidence: `quetzal/deploy.cpp`)
- Chat user → Interactive chat: enters conversation
- Interactive chat → Weight loading: loads weights (evidence: `quetzal/lib/mode/chat.cpp`)
- Interactive chat → Tokenizer binary reader: loads tokenizer data (evidence: `quetzal/lib/mode/chat.cpp`)
- Interactive chat → BBPE encode and decode: encodes and decodes (evidence: `quetzal/lib/mode/chat.cpp`)
- Interactive chat → C++ GPT-2 model: runs generation (evidence: `quetzal/lib/mode/chat.cpp`)
- C++ GPT-2 model → Weight loading: reads parameters
- C++ GPT-2 model → C++ transformer block: uses blocks
- C++ transformer block → C++ attention: owns attention (evidence: `quetzal/include/gpt/transformer.hpp`)
- C++ attention → Rotary embeddings: applies rotations (evidence: `quetzal/lib/gpt/attention.cpp`)
- C++ attention → Linear algebra: uses operations (evidence: `quetzal/lib/gpt/attention.cpp`)
- C++ transformer block → Tensor operations: uses tensors (evidence: `quetzal/include/gpt/transformer.hpp`)
- Interactive chat → Generated text: streams reply (evidence: `quetzal/lib/mode/chat.cpp`)
- Attention inspection → Diagnostic output: shows attention
- Layer performance → Diagnostic output: reports timings
- Tensor experiments → Diagnostic output: writes tensor dumps