How trungtv/pyvi works
Project Type and Purpose
• This is an open-source Python toolkit (“pyvi”) for Vietnamese text processing. It provides:
• Tokenization (splitting text into words)
• Part-of-speech tagging (CRF-based)
• Accent removal and accent restoration
• It ships as a pip-installable library, with model files bundled under pyvi/models and training scripts under pyvi/.
File Structure Analysis
• Top-level package: “pyvi/”
• ViTokenizer.py – implements tokenization
• ViPosTagger.py – implements POS tagging
• ViUtils.py – implements accent removal/adding
• ViDiac.py – likely handles diacritic logic
• models/ – pre-trained CRF models and lookup files
• “data/” – raw training data (dataset.zip, vtb.txt)
• “tests/” – unit tests for tokenizer + test fixtures
• Setup and packaging: setup.py, setup.cfg, MANIFEST.in, release.sh
• README.rst – usage examples, feature list, algorithm notes
Key Architectural Patterns
• Modular Python package: clear separation between tokenization, POS tagging, and utilities
• Model-driven design: core modules load serialized CRF models from disk
• Training pipeline (in notebooks/scripts) is separate from inference code
• Minimal external dependencies (pure Python + python-crfsuite)
Main Components & Interactions
[DEFINE COMPONENTS]
• Client Code (User’s script or interactive prompt)
• ViTokenizer Module
• ViPosTagger Module
• ViUtils Module
• Model Store (pyvi/models directory)
• Data Loader / CRFsuite API (internal to modules)
[DATA FLOW]
1. Client calls ViTokenizer.tokenize(text)
2. ViTokenizer loads CRF model (vidiac.crfsuite), applies features ⇒ token list
3. Client can feed token list into ViPosTagger.postagging(tokens)
4. ViPosTagger loads POS model (pyvipos3.pkl or CRF file) ⇒ tagged tokens
5. Client calls ViUtils.remove_accents(text) or add_accents(token_list)
6. ViUtils uses lookup tables (words.txt) to map characters/words
• Blue for core modules
• Green for data/artifacts (models, data sets)
• Orange for external interfaces (client apps, spaCy)