# himalayaai/nep-tokenizer architecture

> Architecture diagram of the GitHub repository himalayaai/nep-tokenizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/himalayaai/nep-tokenizer
- Repository: https://github.com/himalayaai/nep-tokenizer

## Overview

This repository is a standalone Nepali tokenizer research pipeline: it samples and filters multilingual corpora, trains SentencePiece BPE models, builds fixed evaluation sets, compares tokenizer metrics, and extracts tokenizer notes from research PDFs. The README documents the principal stages; sampled source supports the sampler orchestration, filtering, training, and evaluation edges. Evaluation reporting and paper-note extraction are distinct capabilities. Generated-file locations and some unsampled implementation details are not represented as source paths.

Main entry points: [`scripts/train_sp_bpe.py`](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/train_sp_bpe.py) (SentencePiece trainer), [`scripts/sample_tokenizer_corpus.py`](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/sample_tokenizer_corpus.py) (Corpus sampler), [`scripts/build_eval_corpus.py`](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/build_eval_corpus.py) (Evaluation corpus builder), [`scripts/eval_tokenizer.py`](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/eval_tokenizer.py) (SentencePiece evaluator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_corpus["Corpus preparation"]
  node_parallel_sampling["Parallel sampling"]
  node_sampler["Corpus sampler"]
  node_quality_filters["Quality filters<br/>[quality_filters.py]"]
  node_source_config["Source config"]
  node_group_config["Sampling groups"]
end

subgraph group_training["Tokenizer training"]
  node_training_config["Training config<br/>[train_config.yml]"]
  node_trainer["SentencePiece trainer<br/>[train_sp_bpe.py]"]
  node_tokenizer_models["Tokenizer models"]
end

subgraph group_evaluation["Tokenizer evaluation"]
  node_eval_builder["Evaluation corpus builder"]
  node_eval_config["Evaluation sources<br/>[eval_sources.yml]"]
  node_eval_corpus["Evaluation corpus"]
  node_sp_evaluator["SentencePiece evaluator<br/>[eval_tokenizer.py]"]
  node_hf_evaluator["HF tokenizer evaluator"]
  node_eval_results["Evaluation results"]
  node_table_renderer["Comparison table"]
end

subgraph group_research["Research notes"]
  node_paper_pdfs["Research PDFs"]
  node_paper_extractor["Paper-note extractor"]
  node_paper_notes["Tokenizer notes"]
end

node_user(("Engineer"))
node_hf_datasets[("HF datasets")]
node_hf_hub[("Hugging Face Hub")]

node_user -->|"starts sampling"| node_parallel_sampling
node_parallel_sampling -->|"reads groups"| node_group_config
node_parallel_sampling -->|"launches jobs"| node_sampler
node_sampler -->|"reads sources"| node_source_config
node_sampler -->|"streams rows"| node_hf_datasets
node_sampler -->|"filters text"| node_quality_filters
node_sampler -.->|"uploads shards"| node_hf_hub
node_user -->|"starts training"| node_trainer
node_trainer -->|"reads corpus"| node_hf_hub
node_trainer -->|"reads options"| node_training_config
node_trainer -->|"filters text"| node_quality_filters
node_trainer -->|"writes models"| node_tokenizer_models
node_trainer -.->|"uploads artifacts"| node_hf_hub
node_user -->|"builds corpus"| node_eval_builder
node_eval_builder -->|"reads sources"| node_eval_config
node_eval_builder -->|"streams rows"| node_hf_datasets
node_eval_builder -->|"filters text"| node_quality_filters
node_eval_builder -->|"writes corpus"| node_eval_corpus
node_user -->|"runs evaluation"| node_sp_evaluator
node_sp_evaluator -->|"reads corpus"| node_eval_corpus
node_sp_evaluator -->|"loads models"| node_tokenizer_models
node_sp_evaluator -->|"writes metrics"| node_eval_results
node_user -->|"runs evaluation"| node_hf_evaluator
node_hf_evaluator -->|"reads corpus"| node_eval_corpus
node_hf_evaluator -->|"loads tokenizers"| node_hf_hub
node_hf_evaluator -->|"writes metrics"| node_eval_results
node_user -->|"renders comparison"| node_table_renderer
node_table_renderer -->|"reads metrics"| node_eval_results
node_user -->|"extracts notes"| node_paper_extractor
node_paper_extractor -->|"reads PDFs"| node_paper_pdfs
node_paper_extractor -->|"writes notes"| node_paper_notes

click node_parallel_sampling "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/run_parallel_sampling.py"
click node_sampler "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/sample_tokenizer_corpus.py"
click node_quality_filters "https://github.com/himalayaai/nep-tokenizer/blob/main/nepali_tokenizer/quality_filters.py"
click node_source_config "https://github.com/himalayaai/nep-tokenizer/blob/main/configs/tokenizer_sources.yml"
click node_group_config "https://github.com/himalayaai/nep-tokenizer/blob/main/configs/tokenizer_parallel_groups.yml"
click node_training_config "https://github.com/himalayaai/nep-tokenizer/blob/main/configs/train_config.yml"
click node_trainer "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/train_sp_bpe.py"
click node_eval_builder "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/build_eval_corpus.py"
click node_eval_config "https://github.com/himalayaai/nep-tokenizer/blob/main/configs/eval_sources.yml"
click node_sp_evaluator "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/eval_tokenizer.py"
click node_hf_evaluator "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/eval_tokenizer_hf.py"
click node_table_renderer "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/render_eval_table.py"
click node_paper_extractor "https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/extract_paper_notes.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_parallel_sampling,node_sampler,node_quality_filters,node_source_config,node_group_config toneBlue
class node_training_config,node_trainer,node_tokenizer_models,node_hf_datasets,node_hf_hub toneAmber
class node_eval_builder,node_eval_config,node_eval_corpus,node_sp_evaluator,node_hf_evaluator,node_eval_results,node_table_renderer toneMint
class node_paper_pdfs,node_paper_extractor,node_paper_notes toneRose
class node_user toneIndigo
```

## Components

### Corpus preparation

- [Parallel sampling](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/run_parallel_sampling.py): `scripts/run_parallel_sampling.py`
- [Corpus sampler](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/sample_tokenizer_corpus.py): `scripts/sample_tokenizer_corpus.py`
- [Quality filters](https://github.com/himalayaai/nep-tokenizer/blob/main/nepali_tokenizer/quality_filters.py): `nepali_tokenizer/quality_filters.py`
- [Source config](https://github.com/himalayaai/nep-tokenizer/blob/main/configs/tokenizer_sources.yml): `configs/tokenizer_sources.yml`
- [Sampling groups](https://github.com/himalayaai/nep-tokenizer/blob/main/configs/tokenizer_parallel_groups.yml): `configs/tokenizer_parallel_groups.yml`

### Tokenizer training

- [Training config](https://github.com/himalayaai/nep-tokenizer/blob/main/configs/train_config.yml): `configs/train_config.yml`
- [SentencePiece trainer](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/train_sp_bpe.py): `scripts/train_sp_bpe.py`
- **Tokenizer models**

### Tokenizer evaluation

- [Evaluation corpus builder](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/build_eval_corpus.py): `scripts/build_eval_corpus.py`
- [Evaluation sources](https://github.com/himalayaai/nep-tokenizer/blob/main/configs/eval_sources.yml): `configs/eval_sources.yml`
- **Evaluation corpus**
- [SentencePiece evaluator](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/eval_tokenizer.py): `scripts/eval_tokenizer.py`
- [HF tokenizer evaluator](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/eval_tokenizer_hf.py): `scripts/eval_tokenizer_hf.py`
- **Evaluation results**
- [Comparison table](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/render_eval_table.py): `scripts/render_eval_table.py`

### Research notes

- **Research PDFs**
- [Paper-note extractor](https://github.com/himalayaai/nep-tokenizer/blob/main/scripts/extract_paper_notes.py): `scripts/extract_paper_notes.py`
- **Tokenizer notes**

### Other components

- **Engineer**
- **HF datasets**
- **Hugging Face Hub**

## Connections

- Engineer → Parallel sampling: starts sampling
- Parallel sampling → Sampling groups: reads groups
- Parallel sampling → Corpus sampler: launches jobs
- Corpus sampler → Source config: reads sources
- Corpus sampler → HF datasets: streams rows
- Corpus sampler → Quality filters: filters text
- Corpus sampler → Hugging Face Hub: uploads shards
- Engineer → SentencePiece trainer: starts training
- SentencePiece trainer → Hugging Face Hub: reads corpus
- SentencePiece trainer → Training config: reads options
- SentencePiece trainer → Quality filters: filters text
- SentencePiece trainer → Tokenizer models: writes models
- SentencePiece trainer → Hugging Face Hub: uploads artifacts
- Engineer → Evaluation corpus builder: builds corpus
- Evaluation corpus builder → Evaluation sources: reads sources
- Evaluation corpus builder → HF datasets: streams rows
- Evaluation corpus builder → Quality filters: filters text
- Evaluation corpus builder → Evaluation corpus: writes corpus
- Engineer → SentencePiece evaluator: runs evaluation
- SentencePiece evaluator → Evaluation corpus: reads corpus
- SentencePiece evaluator → Tokenizer models: loads models
- SentencePiece evaluator → Evaluation results: writes metrics
- Engineer → HF tokenizer evaluator: runs evaluation
- HF tokenizer evaluator → Evaluation corpus: reads corpus
- HF tokenizer evaluator → Hugging Face Hub: loads tokenizers
- HF tokenizer evaluator → Evaluation results: writes metrics
- Engineer → Comparison table: renders comparison
- Comparison table → Evaluation results: reads metrics
- Engineer → Paper-note extractor: extracts notes
- Paper-note extractor → Research PDFs: reads PDFs
- Paper-note extractor → Tokenizer notes: writes notes