How chandlerbesse/byte-pair-encoder works
This repository is a small BPE implementation with naive and NumPy-vectorized training and segmentation commands, plus a benchmark/comparison runner. The main workflows read and clean text, learn or load merges, then segment input text; benchmarking runs both implementations and checks their outputs. The graph reflects the supplied scripts and README; the README documents little beyond the project purpose, so capabilities are grounded mainly in sampled source.
Main entry points: bpe_naive.py Naive CLI · bpe_vectorized.py Vectorized CLI · bpe_metrics.py Benchmark runner
Components 20
BPE implementations
- Naive CLI
bpe_naive.py - Text cleaning
bpe_naive.py - Naive trainer
bpe_naive.py - Token merging
bpe_naive.py - Naive segmenter
bpe_naive.py - Vectorized CLI
bpe_vectorized.py - Text cleaning
bpe_vectorized.py - Vectorized trainer
bpe_vectorized.py - Pair counting
bpe_vectorized.py - Array merging
bpe_vectorized.py - Vectorized segmenter
bpe_vectorized.py
Benchmarking
- Metrics CLI
bpe_metrics.py - Benchmark runner
bpe_metrics.py - Implementation adapters
bpe_metrics.py - Output comparison
bpe_metrics.py - Timing and statistics
bpe_metrics.py - JSON result writer
bpe_metrics.py
Other components
- CLI user
- Text files
- Model and results files
Connections 32
- CLI user → Naive CLI: invokes
- CLI user → Vectorized CLI: invokes
- CLI user → Metrics CLI: invokes
- Text files → Naive CLI: supplies text
- Text files → Vectorized CLI: supplies text
- Text files → Metrics CLI: supplies text
- Naive CLI → Text cleaning: cleans text
- Naive CLI → Naive trainer: trains
- Naive CLI → Naive segmenter: segments
- Naive trainer → Token merging: merges pairs
- Naive CLI → Model and results files: writes model
- Model and results files → Naive CLI: provides merges
- Naive CLI → Model and results files: writes segments
- Vectorized CLI → Text cleaning: cleans text
- Vectorized CLI → Vectorized trainer: trains
- Vectorized CLI → Vectorized segmenter: segments
- Vectorized trainer → Pair counting: counts pairs
- Vectorized trainer → Array merging: applies merges
- Vectorized segmenter → Array merging: applies merges
- Vectorized CLI → Model and results files: writes model
- Model and results files → Vectorized CLI: provides model
- Vectorized CLI → Model and results files: writes segments
- Metrics CLI → Benchmark runner: runs benchmark
- Benchmark runner → Implementation adapters: selects methods
- Implementation adapters → Naive trainer: calls
- Implementation adapters → Naive segmenter: calls
- Implementation adapters → Vectorized trainer: calls
- Implementation adapters → Vectorized segmenter: calls
- Benchmark runner → Timing and statistics: measures runs
- Benchmark runner → Output comparison: checks outputs
- Benchmark runner → JSON result writer: saves records
- JSON result writer → Model and results files: writes JSON