How cmiyai/calvin works
This repository combines a byte-pair-encoding tokenizer with custom PyTorch components for a Transformer language model. The tokenizer workflow trains a vocabulary and merge list from text, then encodes text to IDs or decodes IDs back to text. Separately, the model defines embeddings, Transformer blocks, attention, and output layers for language modeling. The excerpts show model composition but do not establish an integration that passes tokenizer output into the model, or a complete training or inference workflow.
Main entry points: model.py Transformer Block
Components 9
BPE Training
- BPE Trainer
tokenizer/bpe_tokenizer.py
Tokenization
- Tokenizer
tokenizer/base_tokenizer.py
Transformer Model
- Transformer LM
model.py - Token Embedding
model.py - Transformer Block
model.py - Self Attention
model.py - Rotary Encoding
model.py - SwiGLU FFN
model.py - Linear and Norm
model.py
Connections 7
- Transformer LM → Token Embedding: constructs
model.py - Transformer LM → Transformer Block: constructs
model.py - Transformer Block → Self Attention: constructs and calls
model.py - Transformer Block → SwiGLU FFN: constructs and calls
model.py - Self Attention → Rotary Encoding: optionally applies
model.py - Self Attention → Linear and Norm: uses projections
model.py - SwiGLU FFN → Linear and Norm: uses layers
model.py