# chandlerbesse/byte-pair-encoder architecture

> Architecture diagram of the GitHub repository chandlerbesse/byte-pair-encoder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/chandlerbesse/byte-pair-encoder
- Repository: https://github.com/chandlerbesse/byte-pair-encoder

## Overview

This repository is a small BPE implementation with naive and NumPy-vectorized training and segmentation commands, plus a benchmark/comparison runner. The main workflows read and clean text, learn or load merges, then segment input text; benchmarking runs both implementations and checks their outputs. The graph reflects the supplied scripts and README; the README documents little beyond the project purpose, so capabilities are grounded mainly in sampled source.

Main entry points: [`bpe_naive.py`](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py) (Naive CLI), [`bpe_vectorized.py`](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py) (Vectorized CLI), [`bpe_metrics.py`](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py) (Benchmark runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_bpe["BPE implementations"]
  node_naive_cli["Naive CLI<br/>[bpe_naive.py]"]
  node_naive_clean["Text cleaning<br/>[bpe_naive.py]"]
  node_naive_train["Naive trainer<br/>[bpe_naive.py]"]
  node_naive_merge["Token merging<br/>[bpe_naive.py]"]
  node_naive_segment["Naive segmenter<br/>[bpe_naive.py]"]
  node_vec_cli["Vectorized CLI<br/>[bpe_vectorized.py]"]
  node_vec_clean["Text cleaning<br/>[bpe_vectorized.py]"]
  node_vec_train["Vectorized trainer<br/>[bpe_vectorized.py]"]
  node_vec_pairs["Pair counting<br/>[bpe_vectorized.py]"]
  node_vec_merge["Array merging<br/>[bpe_vectorized.py]"]
  node_vec_segment["Vectorized segmenter<br/>[bpe_vectorized.py]"]
end

subgraph group_benchmark["Benchmarking"]
  node_metrics_cli["Metrics CLI<br/>[bpe_metrics.py]"]
  node_benchmark_run["Benchmark runner<br/>[bpe_metrics.py]"]
  node_implementations["Implementation adapters<br/>[bpe_metrics.py]"]
  node_compare["Output comparison<br/>[bpe_metrics.py]"]
  node_timing["Timing and statistics<br/>[bpe_metrics.py]"]
  node_save_results["JSON result writer<br/>[bpe_metrics.py]"]
end

node_user(("CLI user"))
node_text["Text files"]
node_artifacts["Model and results files"]

node_user -->|"invokes"| node_naive_cli
node_user -->|"invokes"| node_vec_cli
node_user -->|"invokes"| node_metrics_cli
node_text -->|"supplies text"| node_naive_cli
node_text -->|"supplies text"| node_vec_cli
node_text -->|"supplies text"| node_metrics_cli
node_naive_cli -->|"cleans text"| node_naive_clean
node_naive_cli -->|"trains"| node_naive_train
node_naive_cli -->|"segments"| node_naive_segment
node_naive_train -->|"merges pairs"| node_naive_merge
node_naive_cli -->|"writes model"| node_artifacts
node_artifacts -->|"provides merges"| node_naive_cli
node_naive_cli -->|"writes segments"| node_artifacts
node_vec_cli -->|"cleans text"| node_vec_clean
node_vec_cli -->|"trains"| node_vec_train
node_vec_cli -->|"segments"| node_vec_segment
node_vec_train -->|"counts pairs"| node_vec_pairs
node_vec_train -->|"applies merges"| node_vec_merge
node_vec_segment -->|"applies merges"| node_vec_merge
node_vec_cli -->|"writes model"| node_artifacts
node_artifacts -->|"provides model"| node_vec_cli
node_vec_cli -->|"writes segments"| node_artifacts
node_metrics_cli -->|"runs benchmark"| node_benchmark_run
node_benchmark_run -->|"selects methods"| node_implementations
node_implementations -->|"calls"| node_naive_train
node_implementations -->|"calls"| node_naive_segment
node_implementations -->|"calls"| node_vec_train
node_implementations -->|"calls"| node_vec_segment
node_benchmark_run -->|"measures runs"| node_timing
node_benchmark_run -->|"checks outputs"| node_compare
node_benchmark_run -->|"saves records"| node_save_results
node_save_results -->|"writes JSON"| node_artifacts

click node_naive_cli "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py"
click node_naive_clean "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py"
click node_naive_train "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py"
click node_naive_merge "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py"
click node_naive_segment "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py"
click node_vec_cli "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py"
click node_vec_clean "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py"
click node_vec_train "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py"
click node_vec_pairs "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py"
click node_vec_merge "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py"
click node_vec_segment "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py"
click node_metrics_cli "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py"
click node_benchmark_run "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py"
click node_implementations "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py"
click node_compare "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py"
click node_timing "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py"
click node_save_results "https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_naive_cli,node_naive_clean,node_naive_train,node_naive_merge,node_naive_segment,node_vec_cli,node_vec_clean,node_vec_train,node_vec_pairs,node_vec_merge,node_vec_segment,node_user toneBlue
class node_metrics_cli,node_benchmark_run,node_implementations,node_compare,node_timing,node_save_results toneAmber
class node_text,node_artifacts toneIndigo
```

## Components

### BPE implementations

- [Naive CLI](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py): `bpe_naive.py`
- [Text cleaning](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py): `bpe_naive.py`
- [Naive trainer](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py): `bpe_naive.py`
- [Token merging](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py): `bpe_naive.py`
- [Naive segmenter](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_naive.py): `bpe_naive.py`
- [Vectorized CLI](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py): `bpe_vectorized.py`
- [Text cleaning](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py): `bpe_vectorized.py`
- [Vectorized trainer](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py): `bpe_vectorized.py`
- [Pair counting](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py): `bpe_vectorized.py`
- [Array merging](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py): `bpe_vectorized.py`
- [Vectorized segmenter](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_vectorized.py): `bpe_vectorized.py`

### Benchmarking

- [Metrics CLI](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py): `bpe_metrics.py`
- [Benchmark runner](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py): `bpe_metrics.py`
- [Implementation adapters](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py): `bpe_metrics.py`
- [Output comparison](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py): `bpe_metrics.py`
- [Timing and statistics](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py): `bpe_metrics.py`
- [JSON result writer](https://github.com/chandlerbesse/byte-pair-encoder/blob/main/bpe_metrics.py): `bpe_metrics.py`

### Other components

- **CLI user**
- **Text files**
- **Model and results files**

## Connections

- CLI user → Naive CLI: invokes
- CLI user → Vectorized CLI: invokes
- CLI user → Metrics CLI: invokes
- Text files → Naive CLI: supplies text
- Text files → Vectorized CLI: supplies text
- Text files → Metrics CLI: supplies text
- Naive CLI → Text cleaning: cleans text
- Naive CLI → Naive trainer: trains
- Naive CLI → Naive segmenter: segments
- Naive trainer → Token merging: merges pairs
- Naive CLI → Model and results files: writes model
- Model and results files → Naive CLI: provides merges
- Naive CLI → Model and results files: writes segments
- Vectorized CLI → Text cleaning: cleans text
- Vectorized CLI → Vectorized trainer: trains
- Vectorized CLI → Vectorized segmenter: segments
- Vectorized trainer → Pair counting: counts pairs
- Vectorized trainer → Array merging: applies merges
- Vectorized segmenter → Array merging: applies merges
- Vectorized CLI → Model and results files: writes model
- Model and results files → Vectorized CLI: provides model
- Vectorized CLI → Model and results files: writes segments
- Metrics CLI → Benchmark runner: runs benchmark
- Benchmark runner → Implementation adapters: selects methods
- Implementation adapters → Naive trainer: calls
- Implementation adapters → Naive segmenter: calls
- Implementation adapters → Vectorized trainer: calls
- Implementation adapters → Vectorized segmenter: calls
- Benchmark runner → Timing and statistics: measures runs
- Benchmark runner → Output comparison: checks outputs
- Benchmark runner → JSON result writer: saves records
- JSON result writer → Model and results files: writes JSON