# svarunid/smoltoken architecture

> Architecture diagram of the GitHub repository svarunid/smoltoken, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/svarunid/smoltoken
- Repository: https://github.com/svarunid/smoltoken

## Overview

##### Main Components
- Rust Core Library: This is the heart of the project (see the rust/ directory). It contains the optimized BPE algorithm implementation, multi-threading using the Rayon library, and core functionalities. The source code (rust/src/lib.rs and rust/src/bin/bench.rs) shows both the library and benchmarking binary.
- Python Bindings: Found in the python/ directory, these integrate with the Rust core using PyO3. Files like python/pyproject.toml and modules under python/src/smoltoken (especially core.py and __init__.py) manage the integration, allowing Python users to interact with the tokenization engine.
- Sample Data: The sample/ directory includes example text files (e.g., tinystories.txt) to showcase usage or for benchmarking.
- CI/CD Pipeline: The .github/workflows directory indicates that there are automated build and testing scripts (e.g., build_wheels.yml) ensuring that the library works correctly when publishing Python wheels and Rust crates.
- Build & Configuration Artifacts: Cargo.lock, Cargo.toml, and pyproject.toml provide dependency and configuration management for both language ecosystems.

##### Relationships and Interactions
- The Rust core exposes the main tokenization capabilities. Its functions are directly accessed by the Python bindings.
- The Python layer acts as a wrapper, exposing high-performance tokenization functions to Python users.
- The CI/CD workflows interact with both components to run tests, build the Rust crate, and create Python wheels.
- The sample data provides test cases for validating both the Rust and Python implementations.

##### Architectural Patterns and Design Principles
- Layered Architecture: Different layers exist where the low-level performance-critical code is in Rust, while an abstraction layer in Python provides ease of use.
- Modular Design: The separation between the Rust source (library and binary) and Python bindings (with their own source structure) demonstrates modularity, making it easier to maintain and extend.
- Parallelism: Use of Rayon for multi-threading shows a design geared toward performance and concurrency.
- Integration & Packaging: The use of Cargo for Rust and PyO3 for Python indicates a careful integration strategy between two programming ecosystems.

##### Including Key Technologies
- Rust and its package manager Cargo for building the core.
- Python along with its packaging via pyproject.toml for the bindings.
- Rayon for multi-threading.
- PyO3 for interoperability between Rust and Python.
- GitHub Actions (in .github/workflows) for CI/CD integration.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Build & Configuration Layer
    subgraph "Build & Configuration"
        subgraph "Rust Build Config"
            RCTOML_R["Cargo.toml (Rust)"]:::config
            RCLock_R["Cargo.lock (Rust)"]:::config
        end
        subgraph "Python Build Config"
            Ppy["pyproject.toml"]:::config
            PCtoml["Cargo.toml (Python)"]:::config
            PClock["Cargo.lock (Python)"]:::config
        end
    end

    %% CI/CD Pipeline
    CI["CI/CD Pipeline"]:::cid

    %% Application Layer
    subgraph "Application Layer"
        subgraph "Python Bindings"
            PY["Python API Layer"]:::python
        end
        subgraph "Rust Core Library"
            RC["Tokenization Engine (Rust)"]:::rust
            BM["Benchmark"]:::rust
        end
    end

    %% Sample Data
    subgraph "Sample Data"
        TS["tinystories.txt"]:::data
        Code["code.txt"]:::data
    end

    %% Connections from Build & Configuration to CI/CD Pipeline via config nodes
    Ppy -->|"configures"| CI
    RCTOML_R -->|"configures"| CI

    %% CI/CD Pipeline builds both Python and Rust components
    CI -->|"builds_and_tests"| PY
    CI -->|"builds_and_tests"| RC

    %% Python Bindings call the Rust Core
    PY -->|"calls"| RC

    %% Sample Data used as input for CI/CD testing
    TS -->|"inputs"| CI
    Code -->|"inputs"| CI

    %% Styles
    classDef rust fill:#ffcccc,stroke:#333,stroke-width:2px;
    classDef python fill:#ccffcc,stroke:#333,stroke-width:2px;
    classDef cid fill:#ccccff,stroke:#333,stroke-width:2px;
    classDef data fill:#ffe6cc,stroke:#333,stroke-width:2px;
    classDef config fill:#e6ffff,stroke:#333,stroke-width:2px;

    %% Click Events
    click RC "https://github.com/svarunid/smoltoken/blob/main/rust/src/lib.rs"
    click BM "https://github.com/svarunid/smoltoken/blob/main/rust/src/bin/bench.rs"
    click PY "https://github.com/svarunid/smoltoken/tree/main/python/src/smoltoken"
    click Ppy "https://github.com/svarunid/smoltoken/blob/main/python/pyproject.toml"
    click PCtoml "https://github.com/svarunid/smoltoken/blob/main/python/Cargo.toml"
    click PClock "https://github.com/svarunid/smoltoken/blob/main/python/Cargo.lock"
    click RCTOML_R "https://github.com/svarunid/smoltoken/blob/main/rust/Cargo.toml"
    click RCLock_R "https://github.com/svarunid/smoltoken/blob/main/rust/Cargo.lock"
    click CI "https://github.com/svarunid/smoltoken/blob/main/.github/workflows/build_wheels.yml"
    click TS "https://github.com/svarunid/smoltoken/blob/main/sample/tinystories.txt"
    click Code "https://github.com/svarunid/smoltoken/blob/main/sample/code.txt"
```
