How apdullahyayik/trtokenizer works
1. Project type and boundary
- This repository is a Python library for Turkish tokenization, not an application service.
- Its main runtime boundary is a small public API that accepts text strings and returns tokenized output.
2. Core package
- The trtokenizer/ package is the functional center of the repo.
- trtokenizer/tr_tokenizer.py is the main implementation module and likely defines the public tokenizer classes referenced in the README: SentenceTokenizer and WordTokenizer.
3. Public API surface
- trtokenizer/__init__.py exposes the package as an importable library entry point.
- The intended usage path is: import tokenizer classes from the package, instantiate them, and call tokenize(...) on input text.
4. Sentence tokenization subsystem
- Sentence splitting appears to be the more specialized part of the library.
- The README indicates it uses a curated list of Turkish “non-suffixes” / abbreviations to avoid incorrect sentence breaks after periods.
5. Word tokenization subsystem
- Word tokenization is exposed alongside sentence tokenization but is likely simpler and more regex-driven.
- It operates on already provided text strings and returns word-level tokens.
6. Language rule data
- trtokenizer/tr_non_suffixes is a data file, not code.
- It functions as a language-specific rule list that extends or constrains sentence boundary detection.
- The README states lines starting with # are comments, so this file is meant to be editable by developers without changing code.
7. Error handling
- trtokenizer/errors.py isolates library-specific exceptions.
- This suggests tokenizer failures or invalid inputs are modeled through custom error types rather than generic Python exceptions.
8. Performance layer / compiled artifact
- build/tr_tokenizer.c indicates a generated C/Cython-style build artifact.
- Architecturally, this suggests an optional or historical native acceleration path around the tokenizer logic, separate from the pure Python package code.
- The README also mentions platform-specific shared libraries, reinforcing that performance is part of the distribution story.
9. Packaging and distribution
- setup.py, setup.cfg, and MANIFEST.in define how the library is built and published to PyPI.
- These files control inclusion of non-code assets like tr_non_suffixes and any build outputs needed for installation.
10. Data flow at runtime
- The main flow is: caller provides text → tokenizer class loads language rules/data → regex/rule logic processes text → token list is returned.
- For sentence tokenization, the non-suffix list influences whether punctuation ends a sentence.
- For word tokenization, the flow is more direct and likely centered on token boundary regexes.
11. Key architectural dependency
- The library’s accuracy is driven more by language-specific rule data than by external services.
- There are no signs of network calls, databases, or third-party runtime services in the repository structure.
12. What to draw in an architecture diagram
- Show one box for the Python package API (SentenceTokenizer, WordTokenizer).
- Show a second box for language resources (tr_non_suffixes).
- Show a third box for error types (errors.py).
- Optionally show a separate native/compiled performance layer (build/tr_tokenizer.c) beneath the Python implementation.
- Surround these with packaging/build tooling as the distribution mechanism rather than as part of runtime execution.
Main entry points: tr_tokenizer.py Tokenizer engine · setup.py Setup script · MANIFEST.in Manifest · tr_non_suffixes Non-suffix rules
Components 10
Core library
Public tokenizer API, language rules, and library-specific errors.
- Package API · Importable entry point that exposes the tokenizer classes.
trtokenizer/__init__.py - Tokenizer engine · Main implementation of sentence and word tokenization logic.
trtokenizer/tr_tokenizer.py - Sentence tokenizer · Specialized sentence boundary logic that avoids breaking on abbreviations.
- Word tokenizer · Word-level tokenization for already provided text strings.
- Non-suffix rules · Editable Turkish abbreviation and non-suffix list used by sentence splitting.
trtokenizer/tr_non_suffixes - Library errors · Custom exception types used by the tokenizer library.
trtokenizer/errors.py - Native build · Generated C/Cython-style performance artifact for the tokenizer.
build/tr_tokenizer.c
Packaging
Build and distribution files for installation and published artifacts.
- Setup script · Build and install entry point for the library distribution.
setup.py - Setup config · Supplementary packaging configuration for build and publish behavior.
setup.cfg - Manifest · Controls which non-code files are included in source distributions.
MANIFEST.in
Connections 10
- Package API → Tokenizer engine: exports · Public package entry point exposes the tokenizer implementation.
- Tokenizer engine → Sentence tokenizer: contains · Implementation provides sentence tokenization behavior.
- Tokenizer engine → Word tokenizer: contains · Implementation provides word tokenization behavior.
- Sentence tokenizer → Non-suffix rules: uses · Sentence splitting consults Turkish non-suffix and abbreviation rules.
- Tokenizer engine → Library errors: raises · Library-specific failures are modeled with custom exceptions.
- Tokenizer engine → Native build: accelerates · A compiled artifact suggests an optional or historical native performance path.
- Setup script → Tokenizer engine: packages · Build script installs the main Python implementation.
- Setup script → Manifest: includes · Packaging includes non-code language resources in distributions.
- Setup script → Setup config: configures · Setup configuration contributes to build and publish behavior.
- Manifest → Non-suffix rules: ships · Manifest ensures the language rule file is included in source packages.