# trungtv/pyvi architecture

> Architecture diagram of the GitHub repository trungtv/pyvi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trungtv/pyvi
- Repository: https://github.com/trungtv/pyvi

## Overview

##### Project Type and Purpose
- This is an open-source Python toolkit (“pyvi”) for Vietnamese text processing. It provides:
- Tokenization (splitting text into words)
- Part-of-speech tagging (CRF-based)
- Accent removal and accent restoration
- It ships as a pip-installable library, with model files bundled under pyvi/models and training scripts under pyvi/.

##### File Structure Analysis
- Top-level package: “pyvi/”
- ViTokenizer.py – implements tokenization
- ViPosTagger.py – implements POS tagging
- ViUtils.py – implements accent removal/adding
- ViDiac.py – likely handles diacritic logic
- models/ – pre-trained CRF models and lookup files
- “data/” – raw training data (dataset.zip, vtb.txt)
- “tests/” – unit tests for tokenizer + test fixtures
- Setup and packaging: setup.py, setup.cfg, MANIFEST.in, release.sh
- README.rst – usage examples, feature list, algorithm notes

##### Key Architectural Patterns
- Modular Python package: clear separation between tokenization, POS tagging, and utilities
- Model-driven design: core modules load serialized CRF models from disk
- Training pipeline (in notebooks/scripts) is separate from inference code
- Minimal external dependencies (pure Python + python-crfsuite)

##### Main Components & Interactions
[DEFINE COMPONENTS]
- Client Code (User’s script or interactive prompt)
- ViTokenizer Module
- ViPosTagger Module
- ViUtils Module
- Model Store (pyvi/models directory)
- Data Loader / CRFsuite API (internal to modules)
[DATA FLOW]
1. Client calls ViTokenizer.tokenize(text)
2. ViTokenizer loads CRF model (vidiac.crfsuite), applies features ⇒ token list
3. Client can feed token list into ViPosTagger.postagging(tokens)
4. ViPosTagger loads POS model (pyvipos3.pkl or CRF file) ⇒ tagged tokens
5. Client calls ViUtils.remove_accents(text) or add_accents(token_list)
6. ViUtils uses lookup tables (words.txt) to map characters/words

- Blue for core modules
- Green for data/artifacts (models, data sets)
- Orange for external interfaces (client apps, spaCy)

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Interface
    ClientApp["Client Application"]:::external
    SpaCy["spaCy Adapter (optional)"]:::external
    Packaging["Packaging & Distribution"]:::external

    %% Inference Architecture
    subgraph "Inference Architecture"
        direction TB
        ViTokenizer["ViTokenizer Module"]:::core
        ViPosTagger["ViPosTagger Module"]:::core
        ViUtils["ViUtils Module"]:::core
        ViDiac["Diacritic Logic (ViDiac)"]:::core
        ModelStore["Model Store\n(CRF & Pickle Models)"]:::artifact
    end

    %% Training Architecture
    subgraph "Training Architecture"
        direction TB
        RawData["Raw Training Data"]:::artifact
        TrainScript["train_tokenizer.py"]:::training
        TrainNotebook["train_tokenizer.ipynb"]:::training
    end

    %% Test Harness
    TestHarness["Test Harness\n(tests/)"]:::testing

    %% Inference Data Flow
    ClientApp -->|"tokenize(text)"| ViTokenizer
    ViTokenizer -->|"load_model()"| ModelStore
    ModelStore -->|"CRF model"| ViTokenizer

    ClientApp -->|"postagging(tokens)"| ViPosTagger
    ViPosTagger -->|"load_model()"| ModelStore
    ModelStore -->|"POS model"| ViPosTagger

    ClientApp -->|"remove_accents(text)"| ViUtils
    ClientApp -->|"add_accents(token_list)"| ViUtils
    ViUtils -->|"uses diacritic logic"| ViDiac
    ViUtils -->|"lookup words.txt"| ModelStore

    ClientApp -->|"optional integration"| SpaCy

    %% Training Flow
    RawData --> TrainScript
    RawData --> TrainNotebook
    TrainScript -->|"train tokenizer"| ModelStore
    TrainNotebook -->|"train tokenizer"| ModelStore

    %% Testing
    TestHarness -->|tests tokenization| ViTokenizer
    TestHarness -->|tests utils| ViUtils

    %% Packaging Link
    Packaging --> ClientApp

    %% Click Events
    click ClientApp "https://github.com/trungtv/pyvi/blob/master/setup.py"
    click Packaging "https://github.com/trungtv/pyvi/blob/master/setup.py"
    click ViTokenizer "https://github.com/trungtv/pyvi/blob/master/pyvi/ViTokenizer.py"
    click ViPosTagger "https://github.com/trungtv/pyvi/blob/master/pyvi/ViPosTagger.py"
    click ViUtils "https://github.com/trungtv/pyvi/blob/master/pyvi/ViUtils.py"
    click ViDiac "https://github.com/trungtv/pyvi/blob/master/pyvi/ViDiac.py"
    click ModelStore "https://github.com/trungtv/pyvi/tree/master/pyvi/models/"
    click RawData "https://github.com/trungtv/pyvi/tree/master/data/"
    click TrainScript "https://github.com/trungtv/pyvi/blob/master/pyvi/train_tokenizer.py"
    click TrainNotebook "https://github.com/trungtv/pyvi/blob/master/pyvi/train_tokenizer.ipynb"
    click TestHarness "https://github.com/trungtv/pyvi/tree/master/tests/"

    %% Styles
    classDef core fill:#bbdefb,stroke:#1e88e5,color:#0d47a1;
    classDef artifact fill:#c8e6c9,stroke:#43a047,color:#1b5e20;
    classDef external fill:#ffe0b2,stroke:#fb8c00,color:#e65100;
    classDef training fill:#e1bee7,stroke:#8e24aa,color:#4a148c;
    classDef testing fill:#eeeeee,stroke:#9e9e9e,color:#424242
```
