How nvidia/nemo-text-processing works
NeMo Text Processing provides weighted finite-state grammar-based text normalization (TN) and inverse text normalization (ITN), with grammars for many languages and semiotic classes. The user-facing workflow converts text between spoken-form words and written forms; ITN’s sampled entry point selects language-specific classification and verbalization grammars and uses a token parser. The repository also includes hybrid WFST/language-model rescoring and FST alignment utilities. The tree and README support these capabilities, but most TN and language-grammar implementations were not sampled, so their internal wiring is summarized.
Main entry points: inverse_normalize.py ITN entry point · wfst_lm_rescoring.py WFST-LM rescoring · text_normalization TN grammars · inverse_text_normalization ITN classifiers
Components 13
Text normalization
- TN grammars
nemo_text_processing/text_normalization - TN class rules
nemo_text_processing/text_normalization
Inverse normalization
- ITN entry point
nemo_text_processing/inverse_text_normalization/inverse_normalize.py - ITN classifiers
nemo_text_processing/inverse_text_normalization - Token parser
nemo_text_processing/text_normalization/token_parser.py - ITN verbalizers
nemo_text_processing/inverse_text_normalization
Hybrid rescoring
- WFST-LM rescoring
nemo_text_processing/hybrid/wfst_lm_rescoring.py - Scoring utilities
nemo_text_processing/hybrid/utils.py - MLM scorer
nemo_text_processing/hybrid/mlm_scorer.py
FST utilities
- FST alignment
nemo_text_processing/fst_alignment/alignment.py
Other components
- Text user
- Spoken-form text
- Written-form text
Connections 12
- Text user → TN grammars: submits text
- TN grammars → TN class rules: applies grammars
- TN class rules → Written-form text: produces text
- Text user → ITN entry point: submits text
- ITN entry point → ITN classifiers: selects language rules
- ITN entry point → Token parser: parses tokens
- ITN entry point → ITN verbalizers: verbalizes tokens
- ITN entry point → Spoken-form text: returns text
- Written-form text → WFST-LM rescoring: provides candidates
- WFST-LM rescoring → Scoring utilities: filters candidates
- WFST-LM rescoring → MLM scorer: scores candidates
- Text user → FST alignment: requests alignment