How nawaz0x1/bengali-bpe-tokenizer works
1. What this repo is
- This is a Python package plus CLI for training and using a Bengali-focused Byte Pair Encoding tokenizer.
- The core architecture is a small in-memory text-processing pipeline: corpus text in, learned vocabulary and merge rules out, then encode/decode against that model.
2. Primary runtime boundary
- The package lives under src/bpe/, which is the production code boundary.
- bpe.py at the repo root is the user-facing entry point for CLI commands (train, encode, decode, stats, inspect), while src/bpe/cli.py contains the actual command wiring.
3. Training data flow
- Training starts from a UTF-8 corpus file such as examples/corpus.txt.
- trainer.py builds the BPE model by counting symbol pairs, applying merges, and producing a learned vocabulary/merge table.
- The model is then saved to an output/ directory, which is treated as generated artifact storage rather than source code.
4. Encoding and decoding flow
- tokenizer.py is the high-level facade used by both the CLI and Python API.
- encoder.py turns text into token IDs using the learned vocabulary and merge rules.
- decoder.py maps token IDs back to text, making the tokenizer bidirectional rather than encode-only.
5. Vocabulary and model representation
- vocabulary.py is the boundary for token ↔ ID mapping and model serialization concerns.
- The README implies the saved model includes token inventory plus merge metadata, so downstream encode/decode calls depend on the persisted model directory, not the original corpus.
6. Unicode and Bengali script handling
- unicode_utils.py is a critical architecture piece because the tokenizer operates on Unicode code points, not raw UTF-8 bytes.
- This is what makes the implementation script-safe for Bengali and other non-Latin scripts, preserving combining marks, virama, conjuncts, and NFC-normalized forms.
- That boundary prevents byte-level corruption and is a major differentiator from GPT-style byte BPE.
7. Utility and support layer
- utils.py holds shared non-core helpers such as file I/O, timing, and metrics.
- benchmark.py is a standalone comparison harness against tiktoken, used to measure token counts and compression ratios rather than to serve the tokenizer itself.
- docs/gen_benchmark_plot.py suggests a small analysis pipeline for generating benchmark figures.
8. Research and analysis artifacts
- The fertility_aware_tokenizer/ folder is a research workspace, not the main production package.
- It contains an exploratory notebook, merge summaries, comparison CSVs, and generated figures, indicating a sidecar analysis track for evaluating token fertility and tokenizer quality.
9. Tests as subsystem boundaries
- tests/test_unicode_utils.py, test_vocabulary.py, test_trainer.py, and test_tokenizer.py mirror the main modules, so the test suite validates each architectural layer independently.
- This suggests the intended contract is: Unicode normalization helpers, vocabulary behavior, merge training, and end-to-end tokenization.
10. Tooling and delivery
- pyproject.toml, requirements.txt, .pre-commit-config.yaml, and .github/workflows/ci.yml define the packaging, lint/format discipline, and CI gatekeeping around the package.
- The project is built as a standard installable Python library (src/ layout) with optional developer tooling and reproducible command-line usage.
11. How to draw the architecture
- Draw a simple pipeline: corpus file → Unicode normalization/pre-tokenization → pair counting/merge training → vocabulary + merge artifacts → tokenizer instance → encode/decode operations.
- Surround that with two outer consumers: the CLI entry point for humans and the Python API for library users.
- Treat benchmark.py, notebooks, and docs/ as analysis/validation side paths, not part of the core runtime path.
Main entry points: tokenizer.py Tokenizer · trainer.py Trainer · vocabulary.py Vocabulary · cli.py CLI
Components 18
Runtime
Production package and user-facing entrypoints
- bpe.py · Top-level command entrypoint for train, encode, decode, stats, and inspect.
bpe.py - CLI · Parses commands and connects the user-facing interface to library operations.
src/bpe/cli.py - Tokenizer · High-level API that exposes model-backed tokenization to CLI and library callers.
src/bpe/tokenizer.py
Training
Corpus ingestion and BPE model building
- Trainer · Builds the BPE model by counting pairs, applying merges, and producing learned artifacts.
src/bpe/trainer.py - Unicode utils · Handles Unicode-aware preprocessing for Bengali and other non-Latin scripts.
src/bpe/unicode_utils.py - Corpus · Example UTF-8 corpus used as input for tokenizer training.
examples/corpus.txt
Codec
Encoding and decoding around a learned model
Support
Shared helpers and model representation
- Vocabulary · Defines token inventory, ID assignment, and model serialization boundaries.
src/bpe/vocabulary.py - Utils · Common non-core helpers for file I/O, timing, and metrics.
src/bpe/utils.py - output/ · Directory for saved trained artifacts such as vocabulary and merge metadata.
output/.gitkeep
Analysis
Benchmarking and research sidecar
- Benchmark · Standalone harness for comparing token counts and compression ratios.
benchmark.py - Plot script · Generates benchmark figures from evaluation data.
docs/gen_benchmark_plot.py - Research · Exploratory notebooks, CSV summaries, and generated figures for fertility analysis.
fertility_aware_tokenizer/fertility_research.ipynb
Tests
Unit tests mirroring the main modules
- Unicode tests · Validates Unicode normalization and script-safe text handling.
tests/test_unicode_utils.py - Vocab tests · Validates token inventory and ID mapping behavior.
tests/test_vocabulary.py - Trainer tests · Validates BPE merge learning and model construction.
tests/test_trainer.py - Tokenizer tests · Validates end-to-end tokenization behavior.
tests/test_tokenizer.py
Connections 22
- bpe.py → CLI: invokes · The root CLI entrypoint delegates command handling to the package CLI module.
- CLI → Trainer: train · CLI training commands drive model construction.
- CLI → Tokenizer: encode/decode · CLI runtime uses the tokenizer facade for interactive operations.
- Tokenizer → Unicode utils: normalizes · Tokenization begins with Unicode-aware preprocessing.
- Tokenizer → Encoder: encodes · Facade routes text-to-token conversion to the encoder.
- Tokenizer → Decoder: decodes · Facade routes token-to-text reconstruction to the decoder.
- Trainer → Unicode utils: preprocesses · Training uses the same Unicode-safe text processing boundary.
- Trainer → Vocabulary: builds · Training produces the learned vocabulary and model metadata.
- Trainer → output/: writes · Saved artifacts are stored in the generated model directory.
- Encoder → Vocabulary: reads · Encoding depends on the learned token inventory and IDs.
- Encoder → output/: loads model · Encoder consumes persisted merges and vocabulary from saved artifacts.
- Decoder → Vocabulary: reads · Decoding uses the token-id mapping to recover text.
- Decoder → output/: loads model · Decoder consumes persisted model artifacts.
- Vocabulary → Utils: uses · Vocabulary serialization and file handling rely on shared helpers.
- Benchmark → Tokenizer: measures · Benchmark harness evaluates tokenizer behavior against alternatives.
- Plot script → Benchmark: visualizes · Plot script turns benchmark output into figures.
- Research → Benchmark: analyzes · Research artifacts compare tokenizer performance and fertility.
- Unicode tests → Unicode utils: verifies · Tests cover Unicode handling contracts.
- Vocab tests → Vocabulary: verifies · Tests cover vocabulary behavior.
- Trainer tests → Trainer: verifies · Tests cover model training behavior.
- Tokenizer tests → Tokenizer: verifies · Tests cover end-to-end tokenization behavior.
- Corpus → Trainer: input corpus · Training starts from a UTF-8 corpus file.