# nawaz0x1/bengali-bpe-tokenizer architecture

> Architecture diagram of the GitHub repository nawaz0x1/bengali-bpe-tokenizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nawaz0x1/bengali-bpe-tokenizer
- Repository: https://github.com/nawaz0x1/bengali-bpe-tokenizer

## Overview

1. **What this repo is**
   - This is a Python package plus CLI for training and using a Bengali-focused Byte Pair Encoding tokenizer.
   - The core architecture is a small in-memory text-processing pipeline: corpus text in, learned vocabulary and merge rules out, then encode/decode against that model.

2. **Primary runtime boundary**
   - The package lives under `src/bpe/`, which is the production code boundary.
   - `bpe.py` at the repo root is the user-facing entry point for CLI commands (`train`, `encode`, `decode`, `stats`, `inspect`), while `src/bpe/cli.py` contains the actual command wiring.

3. **Training data flow**
   - Training starts from a UTF-8 corpus file such as `examples/corpus.txt`.
   - `trainer.py` builds the BPE model by counting symbol pairs, applying merges, and producing a learned vocabulary/merge table.
   - The model is then saved to an `output/` directory, which is treated as generated artifact storage rather than source code.

4. **Encoding and decoding flow**
   - `tokenizer.py` is the high-level facade used by both the CLI and Python API.
   - `encoder.py` turns text into token IDs using the learned vocabulary and merge rules.
   - `decoder.py` maps token IDs back to text, making the tokenizer bidirectional rather than encode-only.

5. **Vocabulary and model representation**
   - `vocabulary.py` is the boundary for token ↔ ID mapping and model serialization concerns.
   - The README implies the saved model includes token inventory plus merge metadata, so downstream encode/decode calls depend on the persisted model directory, not the original corpus.

6. **Unicode and Bengali script handling**
   - `unicode_utils.py` is a critical architecture piece because the tokenizer operates on Unicode code points, not raw UTF-8 bytes.
   - This is what makes the implementation script-safe for Bengali and other non-Latin scripts, preserving combining marks, virama, conjuncts, and NFC-normalized forms.
   - That boundary prevents byte-level corruption and is a major differentiator from GPT-style byte BPE.

7. **Utility and support layer**
   - `utils.py` holds shared non-core helpers such as file I/O, timing, and metrics.
   - `benchmark.py` is a standalone comparison harness against `tiktoken`, used to measure token counts and compression ratios rather than to serve the tokenizer itself.
   - `docs/gen_benchmark_plot.py` suggests a small analysis pipeline for generating benchmark figures.

8. **Research and analysis artifacts**
   - The `fertility_aware_tokenizer/` folder is a research workspace, not the main production package.
   - It contains an exploratory notebook, merge summaries, comparison CSVs, and generated figures, indicating a sidecar analysis track for evaluating token fertility and tokenizer quality.

9. **Tests as subsystem boundaries**
   - `tests/test_unicode_utils.py`, `test_vocabulary.py`, `test_trainer.py`, and `test_tokenizer.py` mirror the main modules, so the test suite validates each architectural layer independently.
   - This suggests the intended contract is: Unicode normalization helpers, vocabulary behavior, merge training, and end-to-end tokenization.

10. **Tooling and delivery**
   - `pyproject.toml`, `requirements.txt`, `.pre-commit-config.yaml`, and `.github/workflows/ci.yml` define the packaging, lint/format discipline, and CI gatekeeping around the package.
   - The project is built as a standard installable Python library (`src/` layout) with optional developer tooling and reproducible command-line usage.

11. **How to draw the architecture**
   - Draw a simple pipeline: corpus file → Unicode normalization/pre-tokenization → pair counting/merge training → vocabulary + merge artifacts → tokenizer instance → encode/decode operations.
   - Surround that with two outer consumers: the CLI entry point for humans and the Python API for library users.
   - Treat `benchmark.py`, notebooks, and `docs/` as analysis/validation side paths, not part of the core runtime path.

Main entry points: [`src/bpe/tokenizer.py`](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/tokenizer.py) (Tokenizer), [`src/bpe/trainer.py`](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/trainer.py) (Trainer), [`src/bpe/vocabulary.py`](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/vocabulary.py) (Vocabulary), [`src/bpe/cli.py`](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/cli.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_runtime["Runtime"]
  node_n_bpe_entry["bpe.py<br/>CLI entrypoint<br/>[bpe.py]"]
  node_n_cli["CLI<br/>command wiring<br/>[cli.py]"]
  node_n_tokenizer["Tokenizer<br/>facade<br/>[tokenizer.py]"]
end

subgraph group_grp_train["Training"]
  node_n_trainer["Trainer<br/>BPE training<br/>[trainer.py]"]
  node_n_unicode["Unicode utils<br/>text normalization<br/>[unicode_utils.py]"]
  node_n_examples["Corpus<br/>training data<br/>[corpus.txt]"]
end

subgraph group_grp_codec["Codec"]
  node_n_encoder["Encoder<br/>encode path<br/>[encoder.py]"]
  node_n_decoder["Decoder<br/>decode path<br/>[decoder.py]"]
end

subgraph group_grp_support["Support"]
  node_n_vocab[("Vocabulary<br/>token-id mapping<br/>[vocabulary.py]")]
  node_n_utils["Utils<br/>shared helpers<br/>[utils.py]"]
  node_n_output[("output/<br/>generated model store<br/>[.gitkeep]")]
end

subgraph group_grp_analysis["Analysis"]
  node_n_benchmark["Benchmark<br/>comparison harness<br/>[benchmark.py]"]
  node_n_plot["Plot script<br/>analysis script"]
  node_n_research["Research<br/>analysis workspace"]
end

subgraph group_grp_tests["Tests"]
  node_n_test_unicode["Unicode tests<br/>unit tests"]
  node_n_test_vocab["Vocab tests<br/>unit tests<br/>[test_vocabulary.py]"]
  node_n_test_trainer["Trainer tests<br/>unit tests<br/>[test_trainer.py]"]
  node_n_test_tokenizer["Tokenizer tests<br/>unit tests<br/>[test_tokenizer.py]"]
end

node_n_bpe_entry -->|"invokes"| node_n_cli
node_n_cli -->|"train"| node_n_trainer
node_n_cli -->|"encode/decode"| node_n_tokenizer
node_n_tokenizer -->|"normalizes"| node_n_unicode
node_n_tokenizer -->|"encodes"| node_n_encoder
node_n_tokenizer -->|"decodes"| node_n_decoder
node_n_trainer -->|"preprocesses"| node_n_unicode
node_n_trainer -->|"builds"| node_n_vocab
node_n_trainer -->|"writes"| node_n_output
node_n_encoder -->|"reads"| node_n_vocab
node_n_encoder -->|"loads model"| node_n_output
node_n_decoder -->|"reads"| node_n_vocab
node_n_decoder -->|"loads model"| node_n_output
node_n_vocab -->|"uses"| node_n_utils
node_n_benchmark -->|"measures"| node_n_tokenizer
node_n_plot -->|"visualizes"| node_n_benchmark
node_n_research -->|"analyzes"| node_n_benchmark
node_n_test_unicode -->|"verifies"| node_n_unicode
node_n_test_vocab -->|"verifies"| node_n_vocab
node_n_test_trainer -->|"verifies"| node_n_trainer
node_n_test_tokenizer -->|"verifies"| node_n_tokenizer
node_n_examples -->|"input corpus"| node_n_trainer

click node_n_bpe_entry "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/bpe.py"
click node_n_cli "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/cli.py"
click node_n_tokenizer "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/tokenizer.py"
click node_n_trainer "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/trainer.py"
click node_n_unicode "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/unicode_utils.py"
click node_n_encoder "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/encoder.py"
click node_n_decoder "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/decoder.py"
click node_n_vocab "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/vocabulary.py"
click node_n_utils "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/utils.py"
click node_n_output "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/output/.gitkeep"
click node_n_examples "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/examples/corpus.txt"
click node_n_benchmark "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/benchmark.py"
click node_n_plot "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/docs/gen_benchmark_plot.py"
click node_n_research "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/fertility_aware_tokenizer/fertility_research.ipynb"
click node_n_test_unicode "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_unicode_utils.py"
click node_n_test_vocab "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_vocabulary.py"
click node_n_test_trainer "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_trainer.py"
click node_n_test_tokenizer "https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_tokenizer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_bpe_entry,node_n_cli,node_n_tokenizer toneBlue
class node_n_trainer,node_n_unicode,node_n_examples toneAmber
class node_n_encoder,node_n_decoder toneMint
class node_n_vocab,node_n_utils,node_n_output toneRose
class node_n_benchmark,node_n_plot,node_n_research toneIndigo
class node_n_test_unicode,node_n_test_vocab,node_n_test_trainer,node_n_test_tokenizer toneTeal
```

## Components

### Runtime

Production package and user-facing entrypoints

- [bpe.py](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/bpe.py): CLI entrypoint · Top-level command entrypoint for train, encode, decode, stats, and inspect. · `bpe.py`
- [CLI](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/cli.py): command wiring · Parses commands and connects the user-facing interface to library operations. · `src/bpe/cli.py`
- [Tokenizer](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/tokenizer.py): facade · High-level API that exposes model-backed tokenization to CLI and library callers. · `src/bpe/tokenizer.py`

### Training

Corpus ingestion and BPE model building

- [Trainer](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/trainer.py): BPE training · Builds the BPE model by counting pairs, applying merges, and producing learned artifacts. · `src/bpe/trainer.py`
- [Unicode utils](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/unicode_utils.py): text normalization · Handles Unicode-aware preprocessing for Bengali and other non-Latin scripts. · `src/bpe/unicode_utils.py`
- [Corpus](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/examples/corpus.txt): training data · Example UTF-8 corpus used as input for tokenizer training. · `examples/corpus.txt`

### Codec

Encoding and decoding around a learned model

- [Encoder](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/encoder.py): encode path · Maps normalized text into token IDs using the learned vocabulary and merge rules. · `src/bpe/encoder.py`
- [Decoder](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/decoder.py): decode path · Reconstructs text from token IDs using the persisted model. · `src/bpe/decoder.py`

### Support

Shared helpers and model representation

- [Vocabulary](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/vocabulary.py): token-id mapping · Defines token inventory, ID assignment, and model serialization boundaries. · `src/bpe/vocabulary.py`
- [Utils](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/src/bpe/utils.py): shared helpers · Common non-core helpers for file I/O, timing, and metrics. · `src/bpe/utils.py`
- [output/](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/output/.gitkeep): generated model store · Directory for saved trained artifacts such as vocabulary and merge metadata. · `output/.gitkeep`

### Analysis

Benchmarking and research sidecar

- [Benchmark](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/benchmark.py): comparison harness · Standalone harness for comparing token counts and compression ratios. · `benchmark.py`
- [Plot script](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/docs/gen_benchmark_plot.py): analysis script · Generates benchmark figures from evaluation data. · `docs/gen_benchmark_plot.py`
- [Research](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/fertility_aware_tokenizer/fertility_research.ipynb): analysis workspace · Exploratory notebooks, CSV summaries, and generated figures for fertility analysis. · `fertility_aware_tokenizer/fertility_research.ipynb`

### Tests

Unit tests mirroring the main modules

- [Unicode tests](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_unicode_utils.py): unit tests · Validates Unicode normalization and script-safe text handling. · `tests/test_unicode_utils.py`
- [Vocab tests](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_vocabulary.py): unit tests · Validates token inventory and ID mapping behavior. · `tests/test_vocabulary.py`
- [Trainer tests](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_trainer.py): unit tests · Validates BPE merge learning and model construction. · `tests/test_trainer.py`
- [Tokenizer tests](https://github.com/nawaz0x1/bengali-bpe-tokenizer/blob/main/tests/test_tokenizer.py): unit tests · Validates end-to-end tokenization behavior. · `tests/test_tokenizer.py`

## Connections

- bpe.py → CLI: invokes · The root CLI entrypoint delegates command handling to the package CLI module.
- CLI → Trainer: train · CLI training commands drive model construction.
- CLI → Tokenizer: encode/decode · CLI runtime uses the tokenizer facade for interactive operations.
- Tokenizer → Unicode utils: normalizes · Tokenization begins with Unicode-aware preprocessing.
- Tokenizer → Encoder: encodes · Facade routes text-to-token conversion to the encoder.
- Tokenizer → Decoder: decodes · Facade routes token-to-text reconstruction to the decoder.
- Trainer → Unicode utils: preprocesses · Training uses the same Unicode-safe text processing boundary.
- Trainer → Vocabulary: builds · Training produces the learned vocabulary and model metadata.
- Trainer → output/: writes · Saved artifacts are stored in the generated model directory.
- Encoder → Vocabulary: reads · Encoding depends on the learned token inventory and IDs.
- Encoder → output/: loads model · Encoder consumes persisted merges and vocabulary from saved artifacts.
- Decoder → Vocabulary: reads · Decoding uses the token-id mapping to recover text.
- Decoder → output/: loads model · Decoder consumes persisted model artifacts.
- Vocabulary → Utils: uses · Vocabulary serialization and file handling rely on shared helpers.
- Benchmark → Tokenizer: measures · Benchmark harness evaluates tokenizer behavior against alternatives.
- Plot script → Benchmark: visualizes · Plot script turns benchmark output into figures.
- Research → Benchmark: analyzes · Research artifacts compare tokenizer performance and fertility.
- Unicode tests → Unicode utils: verifies · Tests cover Unicode handling contracts.
- Vocab tests → Vocabulary: verifies · Tests cover vocabulary behavior.
- Trainer tests → Trainer: verifies · Tests cover model training behavior.
- Tokenizer tests → Tokenizer: verifies · Tests cover end-to-end tokenization behavior.
- Corpus → Trainer: input corpus · Training starts from a UTF-8 corpus file.