valkmjolnir/quetzalcoatlus

How valkmjolnir/quetzalcoatlus works

Quetzalcoatlus is an experimental GPT-2 training process and inference engine. Its documented workflow prepares text or conversation data, tokenizes it, trains a base model, fine-tunes it with SFT, and generates text from a checkpoint. A separate C++ inference engine loads exported weights and tokenizer data for chat and diagnostic modes. The graph preserves these distinct paths and the shared model, tokenizer, and tensor layers. Some training and inference modules were not sampled; their documented capabilities are included, but uncertain internal calls are omitted.

Main entry points: chat.cpp Interactive chat · generate.py Python generation · deploy.cpp C++ mode dispatcher · tokenizer.py Python tokenizer and binning

Components 34

Data preparation

JSONL concatenation tool/concat_data_set.py
Pretrain splitting tool/split_pre_train_data.py
SFT splitting tool/split_sft_train_data.py

Python training and inference

Python tokenizer and binning gpt/tokenizer.py
Pretrain data loader gpt/pre_dataloader.py
SFT data loader gpt/sft_dataloader.py
Pretraining gpt/pre_training.py
SFT training gpt/sft_training.py
Python GPT model gpt/gpt.py
Python transformer gpt/transformer.py
Python attention gpt/attn.py
Python generation gpt/generate.py
Device selection gpt/lib/device.py

C++ inference engine

BBPE tokenizer builder quetzal/lib/bbpe/tokenizer.cpp
Tokenizer export quetzal/tokenizer.cpp
C++ mode dispatcher quetzal/deploy.cpp
Interactive chat quetzal/lib/mode/chat.cpp
Attention inspection quetzal/lib/mode/attn.cpp
Layer performance quetzal/lib/mode/performance.cpp
Tensor experiments quetzal/lib/mode/experiment.cpp
Tokenizer binary reader quetzal/lib/bbpe/bin_reader.cpp
BBPE encode and decode quetzal/include/bbpe/tokenizer.hpp

Model and tensor core

Weight loading quetzal/lib/tensor/weights_manager.cpp
C++ GPT-2 model quetzal/lib/gpt/gpt2.cpp
C++ transformer block quetzal/include/gpt/transformer.hpp
C++ attention quetzal/lib/gpt/attention.cpp
Tensor operations quetzal/include/tensor/tensor.hpp
Linear algebra quetzal/include/tensor/linalg.hpp
Rotary embeddings quetzal/include/tensor/rope.hpp

Other components

Training dataset
CLI user
Chat user
Generated text
Diagnostic output

Connections 40

Training dataset → JSONL concatenation: concatenates
JSONL concatenation → Pretrain splitting: prepares text
Training dataset → SFT splitting: splits conversations
Pretrain splitting → Python tokenizer and binning: supplies text
Python tokenizer and binning → Pretrain data loader: writes token bins
Pretrain data loader → Pretraining: provides batches
Pretraining → Python GPT model: trains
SFT splitting → SFT data loader: supplies conversations
SFT data loader → Python tokenizer and binning: encodes text gpt/sft_dataloader.py
SFT data loader → SFT training: provides batches
SFT training → Python GPT model: fine-tunes
CLI user → Python generation: requests generation
Python generation → Python tokenizer and binning: encodes prompts gpt/generate.py
Python generation → Python GPT model: runs inference gpt/generate.py
Python generation → Device selection: selects device gpt/generate.py
Python GPT model → Python transformer: uses blocks gpt/gpt.py
Python transformer → Python attention: uses attention
Python generation → Generated text: returns text
CLI user → BBPE tokenizer builder: builds vocabulary
BBPE tokenizer builder → Tokenizer export: exports tokenizer quetzal/tokenizer.cpp
CLI user → C++ mode dispatcher: selects mode quetzal/deploy.cpp
C++ mode dispatcher → Interactive chat: dispatches chat quetzal/deploy.cpp
C++ mode dispatcher → Attention inspection: dispatches attention quetzal/deploy.cpp
C++ mode dispatcher → Layer performance: dispatches performance quetzal/deploy.cpp
C++ mode dispatcher → Tensor experiments: dispatches experiment quetzal/deploy.cpp
Chat user → Interactive chat: enters conversation
Interactive chat → Weight loading: loads weights quetzal/lib/mode/chat.cpp
Interactive chat → Tokenizer binary reader: loads tokenizer data quetzal/lib/mode/chat.cpp
Interactive chat → BBPE encode and decode: encodes and decodes quetzal/lib/mode/chat.cpp
Interactive chat → C++ GPT-2 model: runs generation quetzal/lib/mode/chat.cpp
C++ GPT-2 model → Weight loading: reads parameters
C++ GPT-2 model → C++ transformer block: uses blocks
C++ transformer block → C++ attention: owns attention quetzal/include/gpt/transformer.hpp
C++ attention → Rotary embeddings: applies rotations quetzal/lib/gpt/attention.cpp
C++ attention → Linear algebra: uses operations quetzal/lib/gpt/attention.cpp
C++ transformer block → Tensor operations: uses tensors quetzal/include/gpt/transformer.hpp
Interactive chat → Generated text: streams reply quetzal/lib/mode/chat.cpp
Attention inspection → Diagnostic output: shows attention
Layer performance → Diagnostic output: reports timings
Tensor experiments → Diagnostic output: writes tensor dumps

More C++ diagrams: czyt1988/saribbon · meiqua/shape_based_matching · intel/compute-runtime · kripken/bananabread · mihailris/voxelcore

Drawn by GitDiagram on October 8, 2026 from the repository's file tree, README and source files. Read as Markdown ·