# openvinotoolkit/openvino_tokenizers architecture

> Architecture diagram of the GitHub repository openvinotoolkit/openvino_tokenizers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openvinotoolkit/openvino_tokenizers
- Repository: https://github.com/openvinotoolkit/openvino_tokenizers

## Overview

Key Components to Include in the System Design:

##### Core Components
- Tokenizer Engine
- Conversion pipeline from HuggingFace models
- Extension System
- Integration with OpenVINO core
- Binary extension architecture
- Model Pipeline
- Demonstrate flow from text input through tokenization to model inference

##### Main Architectural Layers

a. Interface Layer:
- Python API
- C++ API
- CLI Tools
- Clear separation between these interfaces

b. Processing Layer:
- Tokenization Operations
- Detokenization Operations
- Unicode Normalization
- Case Folding
- Special Token Handling

c. Integration Layer:
- OpenVINO Core Integration
- HuggingFace Compatibility
- TensorFlow Text Integration

##### Data Flow Representation

- Input text → Tokenization → Model Processing → Detokenization → Output text
- Model conversion flow: HuggingFace → OpenVINO IR
- Extension loading process

##### Component Details

- Core processing units (rectangles)
- External interfaces (rounded rectangles)

##### Implementation Specifics

Highlight:
- Build system components (CMake)
- ICU integration for text processing
- Platform-specific components
- Test framework integration

Design Guidelines:

##### Metrics
- Supported tokenizer types
- Compatibility information
- Performance characteristics

- The modular nature of the system
- Clear separation of concerns
- Integration points with OpenVINO
- Extensibility mechanisms
- Platform independence

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Interface
        PythonAPI["Python API"]
        CPPAPI["C++ API"]
        CLI["CLI Tools"]
    end

    subgraph TokenizerCore["Tokenizer Engine"]
        Base["Base Tokenizer Interface"]
        BPE["BPE Tokenizer"]
        WordPiece["WordPiece Tokenizer"]
        SentencePiece["SentencePiece Tokenizer"]
        Trie["Trie Tokenizer"]
    end

    subgraph Processing["Processing Layer"]
        Unicode["Unicode Normalization"]
        CaseFold["Case Folding"]
        Regex["Regex Processing"]
        SpecialTokens["Special Token Handling"]
        StringProc["String Processing"]
        TensorOps["Tensor Operations"]
        UTF8["UTF8 Validation"]
    end

    subgraph Integration["Integration Layer"]
        OVIntegration["OpenVINO Integration"]
        HFIntegration["HuggingFace Integration"]
        TFIntegration["TensorFlow Integration"]
    end

    subgraph Pipeline["Model Pipeline"]
        TokenPipeline["Tokenizer Pipeline"]
        ModelConv["Model Conversion"]
    end

    subgraph Build["Build System"]
        CMake["CMake Configuration"]
        Platform["Platform Specifics"]
    end

    subgraph Testing["Testing Framework"]
        Tests["Tests"]
        Benchmark["Benchmark Tools"]
    end

    %% Relationships
    Interface --> TokenizerCore
    TokenizerCore --> Processing
    Processing --> Integration
    Integration --> Pipeline
    TokenizerCore --> Pipeline

    %% Click events
    click PythonAPI "https://github.com/openvinotoolkit/openvino_tokenizers/tree/master/python/openvino_tokenizers"
    click CLI "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/python/openvino_tokenizers/cli.py"
    click Base "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/tokenizer.hpp"
    click BPE "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/bpe_tokenizer.cpp"
    click WordPiece "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/wordpiece_tokenizer.cpp"
    click SentencePiece "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/sentence_piece.cpp"
    click Trie "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/trie_tokenizer.cpp"
    click Unicode "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/normalize_unicode.cpp"
    click CaseFold "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/case_fold.cpp"
    click Regex "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/regex_normalization.cpp"
    click SpecialTokens "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/special_tokens_split.cpp"
    click StringProc "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/string_tensor_pack.cpp"
    click TensorOps "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/ragged_tensor_pack.cpp"
    click UTF8 "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/utf8_validate.cpp"
    click OVIntegration "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/ov_extension.cpp"
    click HFIntegration "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/python/openvino_tokenizers/hf_parser.py"
    click TFIntegration "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/src/tensorflow_translators.cpp"
    click TokenPipeline "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/python/openvino_tokenizers/tokenizer_pipeline.py"
    click ModelConv "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/python/openvino_tokenizers/convert_tokenizer.py"
    click CMake "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/CMakeLists.txt"
    click Platform "https://github.com/openvinotoolkit/openvino_tokenizers/blob/master/cmake/platforms.cmake"
    click Tests "https://github.com/openvinotoolkit/openvino_tokenizers/tree/master/tests"
    click Benchmark "https://github.com/openvinotoolkit/openvino_tokenizers/tree/master/benchmark"

    classDef interface fill:#90EE90
    classDef core fill:#87CEEB
    classDef processing fill:#FFE4B5
    classDef integration fill:#DDA0DD
    classDef pipeline fill:#98FB98
    classDef build fill:#D3D3D3
    classDef testing fill:#F0E68C

    class PythonAPI,CPPAPI,CLI interface
    class Base,BPE,WordPiece,SentencePiece,Trie core
    class Unicode,CaseFold,Regex,SpecialTokens,StringProc,TensorOps,UTF8 processing
    class OVIntegration,HFIntegration,TFIntegration integration
    class TokenPipeline,ModelConv pipeline
    class CMake,Platform build
    class Tests,Benchmark testing
```
