# huggingface/tokenizers architecture

> Architecture diagram of the GitHub repository huggingface/tokenizers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/tokenizers
- Repository: https://github.com/huggingface/tokenizers

## Overview

##### Main Components
- Core Tokenizer Library (Rust): This is the backbone of the project. It contains the implementation of tokenizers (models, decoders, normalizers, pre-tokenizers, processors, trainers, etc.) organized mainly inside the “tokenizers/src” directory with related subdirectories (for decoders, models, pre-tokenizers, and so on). Also include the benchmarks and examples.
- Language Bindings:
- Python Binding: Found under “bindings/python”. It wraps the core Rust functionality in a Python interface (including py_src, tests, and examples).
- Node.js Binding: Located under “bindings/node”. It provides a Node.js/JavaScript layer on top of the Rust core.
- Ruby Binding (external): Mention the existence of the Ruby binding (provided externally) for completeness.
- Documentation & Examples: The “docs” folder contains source files and builders (using tools such as Sphinx or a custom solution) to generate detailed API documentation, quicktours, training guides, and tutorials.
- CI/CD and Build Infrastructure: The “.github” directory shows several workflows (for Rust, Python, Node.js releases, documentation, testing, and more). Also note build scripts (Makefiles, Cargo.toml files, etc.) that coordinate the build, test, and release processes.

##### Relationships and Interactions
- The Core library is a self-contained module offering APIs for tokenization, training, and related tasks.
- The CI/CD pipelines (GitHub Workflows) interact with every component: triggering builds/tests for the core implementation and each binding, and updating documentation.
- Documentation tools/processes (located in “docs” and “bindings/node/examples/documentation”, “bindings/python/examples” etc.) use the source code and tests to build user-facing documentation and quicktours.

##### Architectural Patterns and Technology
- Layered Architecture: The design separates concerns between the core implementation and its language-specific interfaces.
- Build and Automation: Integration with multiple CI/CD pipelines, with dedicated workflows for Rust, Python, and Node.js. This shows a strong emphasis on continuous integration and automated testing.
- Plugin/Adapter-like Model: Although not explicitly a plugin architecture, the way bindings are implemented acts as adapters over the common tokenization logic in Rust.

##### Key Technologies
- Rust: The core system, high-performance tokenization library.
- Python & Node.js: High-level language bindings that provide easier adoption by various user communities.
- GitHub Actions: Used extensively for running tests, building documents, releasing new versions, and more.
- Documentation Tools: Sphinx or custom Markdown/MDX-based documentation generators, as seen in the “docs” folder.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CI/CD and Documentation
    CICD["CI/CD & Build Infrastructure"]:::cicd
    DOCS["Documentation & Examples"]:::documentation

    %% Core Library Subgraph
    subgraph "Core Library"
        CORE["Core Tokenizer Library (Rust)"]:::core
        SRC["Source (tokenizers/src)"]:::core
        BENCH["Benchmarks (tokenizers/benches)"]:::core
        CORE_EX["Examples (tokenizers/examples)"]:::core
        CORE --> SRC
        CORE --> BENCH
        CORE --> CORE_EX
    end

    %% Language Bindings Subgraph
    subgraph "Language Bindings"
        PY["Python Binding"]:::bindings
        NODE["Node.js Binding"]:::bindings
        RUBY["Ruby Binding (External)"]:::bindings
    end

    %% Relationships
    PY -->|"calls"| CORE
    NODE -->|"calls"| CORE
    RUBY -->|"calls"| CORE

    CICD -->|"builds/tests"| CORE
    CICD -->|"builds/tests"| PY
    CICD -->|"builds/tests"| NODE
    CICD -->|"updates"| DOCS

    DOCS -->|"generatedFrom"| SRC
    DOCS -->|"generatedFrom"| PY
    DOCS -->|"generatedFrom"| NODE

    %% Click Events
    click SRC "https://github.com/huggingface/tokenizers/tree/main/tokenizers/src"
    click BENCH "https://github.com/huggingface/tokenizers/tree/main/tokenizers/benches"
    click CORE_EX "https://github.com/huggingface/tokenizers/tree/main/tokenizers/examples"
    click PY "https://github.com/huggingface/tokenizers/tree/main/bindings/python"
    click NODE "https://github.com/huggingface/tokenizers/tree/main/bindings/node"
    click DOCS "https://github.com/huggingface/tokenizers/tree/main/docs"
    click CICD "https://github.com/huggingface/tokenizers/blob/main/.github"

    %% Styles
    classDef core fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef bindings fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef cicd fill:#ffa500,stroke:#000,stroke-width:2px;
    classDef documentation fill:#dda0dd,stroke:#000,stroke-width:2px;
```
