# apdullahyayik/trtokenizer architecture

> Architecture diagram of the GitHub repository apdullahyayik/trtokenizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apdullahyayik/trtokenizer
- Repository: https://github.com/apdullahyayik/trtokenizer

## Overview

1. **Project type and boundary**
   - This repository is a **Python library** for Turkish tokenization, not an application service.
   - Its main runtime boundary is a small public API that accepts text strings and returns tokenized output.

2. **Core package**
   - The `trtokenizer/` package is the functional center of the repo.
   - `trtokenizer/tr_tokenizer.py` is the main implementation module and likely defines the public tokenizer classes referenced in the README: `SentenceTokenizer` and `WordTokenizer`.

3. **Public API surface**
   - `trtokenizer/__init__.py` exposes the package as an importable library entry point.
   - The intended usage path is: import tokenizer classes from the package, instantiate them, and call `tokenize(...)` on input text.

4. **Sentence tokenization subsystem**
   - Sentence splitting appears to be the more specialized part of the library.
   - The README indicates it uses a curated list of Turkish “non-suffixes” / abbreviations to avoid incorrect sentence breaks after periods.

5. **Word tokenization subsystem**
   - Word tokenization is exposed alongside sentence tokenization but is likely simpler and more regex-driven.
   - It operates on already provided text strings and returns word-level tokens.

6. **Language rule data**
   - `trtokenizer/tr_non_suffixes` is a data file, not code.
   - It functions as a language-specific rule list that extends or constrains sentence boundary detection.
   - The README states lines starting with `#` are comments, so this file is meant to be editable by developers without changing code.

7. **Error handling**
   - `trtokenizer/errors.py` isolates library-specific exceptions.
   - This suggests tokenizer failures or invalid inputs are modeled through custom error types rather than generic Python exceptions.

8. **Performance layer / compiled artifact**
   - `build/tr_tokenizer.c` indicates a generated C/Cython-style build artifact.
   - Architecturally, this suggests an optional or historical **native acceleration path** around the tokenizer logic, separate from the pure Python package code.
   - The README also mentions platform-specific shared libraries, reinforcing that performance is part of the distribution story.

9. **Packaging and distribution**
   - `setup.py`, `setup.cfg`, and `MANIFEST.in` define how the library is built and published to PyPI.
   - These files control inclusion of non-code assets like `tr_non_suffixes` and any build outputs needed for installation.

10. **Data flow at runtime**
   - The main flow is: **caller provides text → tokenizer class loads language rules/data → regex/rule logic processes text → token list is returned**.
   - For sentence tokenization, the non-suffix list influences whether punctuation ends a sentence.
   - For word tokenization, the flow is more direct and likely centered on token boundary regexes.

11. **Key architectural dependency**
   - The library’s accuracy is driven more by **language-specific rule data** than by external services.
   - There are no signs of network calls, databases, or third-party runtime services in the repository structure.

12. **What to draw in an architecture diagram**
   - Show one box for the **Python package API** (`SentenceTokenizer`, `WordTokenizer`).
   - Show a second box for **language resources** (`tr_non_suffixes`).
   - Show a third box for **error types** (`errors.py`).
   - Optionally show a separate **native/compiled performance layer** (`build/tr_tokenizer.c`) beneath the Python implementation.
   - Surround these with **packaging/build tooling** as the distribution mechanism rather than as part of runtime execution.

Main entry points: [`trtokenizer/tr_tokenizer.py`](https://github.com/apdullahyayik/trtokenizer/blob/master/trtokenizer/tr_tokenizer.py) (Tokenizer engine), [`setup.py`](https://github.com/apdullahyayik/trtokenizer/blob/master/setup.py) (Setup script), [`MANIFEST.in`](https://github.com/apdullahyayik/trtokenizer/blob/master/MANIFEST.in) (Manifest), [`trtokenizer/tr_non_suffixes`](https://github.com/apdullahyayik/trtokenizer/tree/master/trtokenizer/tr_non_suffixes) (Non-suffix rules)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_core["Core library"]
  node_node_api["Package API<br/>Python package<br/>[__init__.py]"]
  node_node_tokenizer["Tokenizer engine<br/>Python implementation<br/>[tr_tokenizer.py]"]
  node_node_sentence["Sentence tokenizer<br/>sentence splitter"]
  node_node_word["Word tokenizer<br/>word splitter"]
  node_node_rules["Non-suffix rules<br/>language data"]
  node_node_errors["Library errors<br/>exceptions<br/>[errors.py]"]
  node_node_native{{"Native build<br/>compiled artifact<br/>[tr_tokenizer.c]"}}
end

subgraph group_group_dist["Packaging"]
  node_node_setup["Setup script<br/>packaging<br/>[setup.py]"]
  node_node_cfg["Setup config<br/>packaging config<br/>[setup.cfg]"]
  node_node_manifest["Manifest<br/>[MANIFEST.in]"]
end

node_node_api -->|"exports"| node_node_tokenizer
node_node_tokenizer -->|"contains"| node_node_sentence
node_node_tokenizer -->|"contains"| node_node_word
node_node_sentence -->|"uses"| node_node_rules
node_node_tokenizer -->|"raises"| node_node_errors
node_node_tokenizer -.->|"accelerates"| node_node_native
node_node_setup -->|"packages"| node_node_tokenizer
node_node_setup -->|"includes"| node_node_manifest
node_node_setup -->|"configures"| node_node_cfg
node_node_manifest -->|"ships"| node_node_rules

click node_node_api "https://github.com/apdullahyayik/trtokenizer/blob/master/trtokenizer/__init__.py"
click node_node_tokenizer "https://github.com/apdullahyayik/trtokenizer/blob/master/trtokenizer/tr_tokenizer.py"
click node_node_rules "https://github.com/apdullahyayik/trtokenizer/tree/master/trtokenizer/tr_non_suffixes"
click node_node_errors "https://github.com/apdullahyayik/trtokenizer/blob/master/trtokenizer/errors.py"
click node_node_native "https://github.com/apdullahyayik/trtokenizer/blob/master/build/tr_tokenizer.c"
click node_node_setup "https://github.com/apdullahyayik/trtokenizer/blob/master/setup.py"
click node_node_cfg "https://github.com/apdullahyayik/trtokenizer/blob/master/setup.cfg"
click node_node_manifest "https://github.com/apdullahyayik/trtokenizer/blob/master/MANIFEST.in"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_api,node_node_tokenizer,node_node_sentence,node_node_word,node_node_rules,node_node_errors,node_node_native toneBlue
class node_node_setup,node_node_cfg,node_node_manifest toneAmber
```

## Components

### Core library

Public tokenizer API, language rules, and library-specific errors.

- [Package API](https://github.com/apdullahyayik/trtokenizer/blob/master/trtokenizer/__init__.py): Python package · Importable entry point that exposes the tokenizer classes. · `trtokenizer/__init__.py`
- [Tokenizer engine](https://github.com/apdullahyayik/trtokenizer/blob/master/trtokenizer/tr_tokenizer.py): Python implementation · Main implementation of sentence and word tokenization logic. · `trtokenizer/tr_tokenizer.py`
- **Sentence tokenizer**: sentence splitter · Specialized sentence boundary logic that avoids breaking on abbreviations.
- **Word tokenizer**: word splitter · Word-level tokenization for already provided text strings.
- [Non-suffix rules](https://github.com/apdullahyayik/trtokenizer/tree/master/trtokenizer/tr_non_suffixes): language data · Editable Turkish abbreviation and non-suffix list used by sentence splitting. · `trtokenizer/tr_non_suffixes`
- [Library errors](https://github.com/apdullahyayik/trtokenizer/blob/master/trtokenizer/errors.py): exceptions · Custom exception types used by the tokenizer library. · `trtokenizer/errors.py`
- [Native build](https://github.com/apdullahyayik/trtokenizer/blob/master/build/tr_tokenizer.c): compiled artifact · Generated C/Cython-style performance artifact for the tokenizer. · `build/tr_tokenizer.c`

### Packaging

Build and distribution files for installation and published artifacts.

- [Setup script](https://github.com/apdullahyayik/trtokenizer/blob/master/setup.py): packaging · Build and install entry point for the library distribution. · `setup.py`
- [Setup config](https://github.com/apdullahyayik/trtokenizer/blob/master/setup.cfg): packaging config · Supplementary packaging configuration for build and publish behavior. · `setup.cfg`
- [Manifest](https://github.com/apdullahyayik/trtokenizer/blob/master/MANIFEST.in): package manifest · Controls which non-code files are included in source distributions. · `MANIFEST.in`

## Connections

- Package API → Tokenizer engine: exports · Public package entry point exposes the tokenizer implementation.
- Tokenizer engine → Sentence tokenizer: contains · Implementation provides sentence tokenization behavior.
- Tokenizer engine → Word tokenizer: contains · Implementation provides word tokenization behavior.
- Sentence tokenizer → Non-suffix rules: uses · Sentence splitting consults Turkish non-suffix and abbreviation rules.
- Tokenizer engine → Library errors: raises · Library-specific failures are modeled with custom exceptions.
- Tokenizer engine → Native build: accelerates · A compiled artifact suggests an optional or historical native performance path.
- Setup script → Tokenizer engine: packages · Build script installs the main Python implementation.
- Setup script → Manifest: includes · Packaging includes non-code language resources in distributions.
- Setup script → Setup config: configures · Setup configuration contributes to build and publish behavior.
- Manifest → Non-suffix rules: ships · Manifest ensures the language rule file is included in source packages.
