How heyavijitroy/tea-benchmark works
TEA is a reproducible multilingual tokenization-equity benchmark. A researcher runs Bengali quality classification on the 120-item corpus, tokenizes six language versions with three tokenizer families, computes per-item TFR and aggregate cost/context metrics, then generates tables and publication figures. The graph separates corpus validation, tokenization, metric outputs, and visualization. It is based on the README and sampled scripts; unsampled implementation details and undocumented wiring are omitted.
Components 19
Corpus Validation
- Source Corpus
data/tea_corpus.csv - Quality Checker
scripts/01_quality_check.py - Flagged Corpus
data/tea_corpus_flagged.csv - Quality Report
data/quality_report.txt
Tokenization
- Tokenization Pipeline
scripts/02_tokenize.py - GPT Tokenizer
scripts/02_tokenize.py - Qwen Tokenizer
scripts/02_tokenize.py - Mistral Tokenizer
scripts/02_tokenize.py
Metric Outputs
- Raw Token Counts
results/raw_token_counts.csv - Item TFR Results
results/tfr_by_item.csv - Summary Statistics
results/summary_stats.csv - Cost Table
results/acm_table.csv - Context Table
results/ecw_table.csv
Visualization
- Figure Generator
scripts/generate_tfr_figure.py - Benchmark Figures
figures - Split Explorer
scripts/token_split_table.py - Split Examples
results/token_split_T1-10.csv
Other components
- Researcher
- HuggingFace Registry
Connections 24
- Researcher → Quality Checker: runs
- Researcher → Tokenization Pipeline: runs
- Researcher → Figure Generator: runs
- Source Corpus → Quality Checker: reads
- Quality Checker → Flagged Corpus: writes
- Quality Checker → Quality Report: writes
- Flagged Corpus → Tokenization Pipeline: reads
- Tokenization Pipeline → GPT Tokenizer: loads
- Tokenization Pipeline → Qwen Tokenizer: loads
- Tokenization Pipeline → Mistral Tokenizer: loads
- Qwen Tokenizer → HuggingFace Registry: downloads tokenizer
- Mistral Tokenizer → HuggingFace Registry: downloads tokenizer
- Tokenization Pipeline → Raw Token Counts: writes
- Tokenization Pipeline → Item TFR Results: writes
- Raw Token Counts → Summary Statistics: aggregates
- Item TFR Results → Summary Statistics: aggregates
- Raw Token Counts → Cost Table: computes cost
- Item TFR Results → Cost Table: computes cost
- Item TFR Results → Context Table: computes context
- Summary Statistics → Figure Generator: reads
- Context Table → Figure Generator: reads
- Figure Generator → Benchmark Figures: writes
- Source Corpus → Split Explorer: reads
- Split Explorer → Split Examples: writes