How himalayaai/nep-tokenizer works
This repository is a standalone Nepali tokenizer research pipeline: it samples and filters multilingual corpora, trains SentencePiece BPE models, builds fixed evaluation sets, compares tokenizer metrics, and extracts tokenizer notes from research PDFs. The README documents the principal stages; sampled source supports the sampler orchestration, filtering, training, and evaluation edges. Evaluation reporting and paper-note extraction are distinct capabilities. Generated-file locations and some unsampled implementation details are not represented as source paths.
Main entry points: train_sp_bpe.py SentencePiece trainer · sample_tokenizer_corpus.py Corpus sampler · build_eval_corpus.py Evaluation corpus builder · eval_tokenizer.py SentencePiece evaluator
Components 21
Corpus preparation
- Parallel sampling
scripts/run_parallel_sampling.py - Corpus sampler
scripts/sample_tokenizer_corpus.py - Quality filters
nepali_tokenizer/quality_filters.py - Source config
configs/tokenizer_sources.yml - Sampling groups
configs/tokenizer_parallel_groups.yml
Tokenizer training
- Training config
configs/train_config.yml - SentencePiece trainer
scripts/train_sp_bpe.py - Tokenizer models
Tokenizer evaluation
- Evaluation corpus builder
scripts/build_eval_corpus.py - Evaluation sources
configs/eval_sources.yml - Evaluation corpus
- SentencePiece evaluator
scripts/eval_tokenizer.py - HF tokenizer evaluator
scripts/eval_tokenizer_hf.py - Evaluation results
- Comparison table
scripts/render_eval_table.py
Research notes
- Research PDFs
- Paper-note extractor
scripts/extract_paper_notes.py - Tokenizer notes
Other components
- Engineer
- HF datasets
- Hugging Face Hub
Connections 31
- Engineer → Parallel sampling: starts sampling
- Parallel sampling → Sampling groups: reads groups
- Parallel sampling → Corpus sampler: launches jobs
- Corpus sampler → Source config: reads sources
- Corpus sampler → HF datasets: streams rows
- Corpus sampler → Quality filters: filters text
- Corpus sampler → Hugging Face Hub: uploads shards
- Engineer → SentencePiece trainer: starts training
- SentencePiece trainer → Hugging Face Hub: reads corpus
- SentencePiece trainer → Training config: reads options
- SentencePiece trainer → Quality filters: filters text
- SentencePiece trainer → Tokenizer models: writes models
- SentencePiece trainer → Hugging Face Hub: uploads artifacts
- Engineer → Evaluation corpus builder: builds corpus
- Evaluation corpus builder → Evaluation sources: reads sources
- Evaluation corpus builder → HF datasets: streams rows
- Evaluation corpus builder → Quality filters: filters text
- Evaluation corpus builder → Evaluation corpus: writes corpus
- Engineer → SentencePiece evaluator: runs evaluation
- SentencePiece evaluator → Evaluation corpus: reads corpus
- SentencePiece evaluator → Tokenizer models: loads models
- SentencePiece evaluator → Evaluation results: writes metrics
- Engineer → HF tokenizer evaluator: runs evaluation
- HF tokenizer evaluator → Evaluation corpus: reads corpus
- HF tokenizer evaluator → Hugging Face Hub: loads tokenizers
- HF tokenizer evaluator → Evaluation results: writes metrics
- Engineer → Comparison table: renders comparison
- Comparison table → Evaluation results: reads metrics
- Engineer → Paper-note extractor: extracts notes
- Paper-note extractor → Research PDFs: reads PDFs
- Paper-note extractor → Tokenizer notes: writes notes