# cmiyai/calvin architecture

> Architecture diagram of the GitHub repository cmiyai/calvin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cmiyai/calvin
- Repository: https://github.com/cmiyai/calvin

## Overview

This repository combines a byte-pair-encoding tokenizer with custom PyTorch components for a Transformer language model. The tokenizer workflow trains a vocabulary and merge list from text, then encodes text to IDs or decodes IDs back to text. Separately, the model defines embeddings, Transformer blocks, attention, and output layers for language modeling. The excerpts show model composition but do not establish an integration that passes tokenizer output into the model, or a complete training or inference workflow.

Main entry points: [`model.py`](https://github.com/cmiyai/calvin/blob/main/model.py) (Transformer Block)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_bpe_training["BPE Training"]
  node_bpe_trainer["BPE Trainer<br/>[bpe_tokenizer.py]"]
end

subgraph group_tokenization["Tokenization"]
  node_tokenizer["Tokenizer<br/>[base_tokenizer.py]"]
end

subgraph group_transformer["Transformer Model"]
  node_transformer_lm["Transformer LM<br/>[model.py]"]
  node_embedding["Token Embedding<br/>[model.py]"]
  node_transformer_block["Transformer Block<br/>[model.py]"]
  node_attention["Self Attention<br/>[model.py]"]
  node_rope["Rotary Encoding<br/>[model.py]"]
  node_swiglu["SwiGLU FFN<br/>[model.py]"]
  node_linear_norm["Linear and Norm<br/>[model.py]"]
end

node_transformer_lm -->|"constructs"| node_embedding
node_transformer_lm -->|"constructs"| node_transformer_block
node_transformer_block -->|"constructs and calls"| node_attention
node_transformer_block -->|"constructs and calls"| node_swiglu
node_attention -.->|"optionally applies"| node_rope
node_attention -->|"uses projections"| node_linear_norm
node_swiglu -->|"uses layers"| node_linear_norm

click node_bpe_trainer "https://github.com/cmiyai/calvin/blob/main/tokenizer/bpe_tokenizer.py"
click node_tokenizer "https://github.com/cmiyai/calvin/blob/main/tokenizer/base_tokenizer.py"
click node_transformer_lm "https://github.com/cmiyai/calvin/blob/main/model.py"
click node_embedding "https://github.com/cmiyai/calvin/blob/main/model.py"
click node_transformer_block "https://github.com/cmiyai/calvin/blob/main/model.py"
click node_attention "https://github.com/cmiyai/calvin/blob/main/model.py"
click node_rope "https://github.com/cmiyai/calvin/blob/main/model.py"
click node_swiglu "https://github.com/cmiyai/calvin/blob/main/model.py"
click node_linear_norm "https://github.com/cmiyai/calvin/blob/main/model.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bpe_trainer toneBlue
class node_tokenizer toneAmber
class node_transformer_lm,node_embedding,node_transformer_block,node_attention,node_rope,node_swiglu,node_linear_norm toneMint
```

## Components

### BPE Training

- [BPE Trainer](https://github.com/cmiyai/calvin/blob/main/tokenizer/bpe_tokenizer.py): `tokenizer/bpe_tokenizer.py`

### Tokenization

- [Tokenizer](https://github.com/cmiyai/calvin/blob/main/tokenizer/base_tokenizer.py): `tokenizer/base_tokenizer.py`

### Transformer Model

- [Transformer LM](https://github.com/cmiyai/calvin/blob/main/model.py): `model.py`
- [Token Embedding](https://github.com/cmiyai/calvin/blob/main/model.py): `model.py`
- [Transformer Block](https://github.com/cmiyai/calvin/blob/main/model.py): `model.py`
- [Self Attention](https://github.com/cmiyai/calvin/blob/main/model.py): `model.py`
- [Rotary Encoding](https://github.com/cmiyai/calvin/blob/main/model.py): `model.py`
- [SwiGLU FFN](https://github.com/cmiyai/calvin/blob/main/model.py): `model.py`
- [Linear and Norm](https://github.com/cmiyai/calvin/blob/main/model.py): `model.py`

## Connections

- Transformer LM → Token Embedding: constructs (evidence: `model.py`)
- Transformer LM → Transformer Block: constructs (evidence: `model.py`)
- Transformer Block → Self Attention: constructs and calls (evidence: `model.py`)
- Transformer Block → SwiGLU FFN: constructs and calls (evidence: `model.py`)
- Self Attention → Rotary Encoding: optionally applies (evidence: `model.py`)
- Self Attention → Linear and Norm: uses projections (evidence: `model.py`)
- SwiGLU FFN → Linear and Norm: uses layers (evidence: `model.py`)
