# kojix2/tiktoken-cr architecture

> Architecture diagram of the GitHub repository kojix2/tiktoken-cr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kojix2/tiktoken-cr
- Repository: https://github.com/kojix2/tiktoken-cr

## Overview

##### Project type and purpose
- From the README and file tree, this is not a web-app or full-stack system but a language-binding library (“shard”) for Crystal.
- It wraps/links to an external Rust/C implementation (“tiktoken-c”) to provide GPT tokenization to Crystal programs.

##### File structure
- Top-level:
- shard.yml, Makefile, README.md, LICENSE, .gitignore, .github/ (CI/workflows)
- src/ – Crystal wrapper code (tiktoken.cr plus a tiktoken/ subfolder)
- tiktoken-c/ – Git submodule containing the Rust source built as a dynamic library via Cargo
- spec/ – Crystal unit tests against both the pure-Crystal stubs and the underlying C library
- examples/ – sample scripts showing how end users call the API
- Key patterns:
- FFI boundary: Crystal code in src/tiktoken/libtiktoken.cr calls into the compiled tiktoken-c dynamic library.
- “Core” logic sits in the Rust crate; Crystal provides idiomatic API & type definitions.

##### README for additional insights
- Installation steps show build flow:
Crystal program → requires “tiktoken” shard → at compile time links to tiktoken-c via LD_LIBRARY_PATH or --link-flags.
- Usage snippets reveal the two API layers:
- Tiktoken.encoding_for_model / Tiktoken::Encoding.* as the Crystal interface
- encode/decode methods that trigger Rust library functionality

## Diagram (Mermaid)

```mermaid
graph TB
    %% Consumer Layer
    subgraph "Consumer Layer"
        App["Your Crystal Application"]:::crystal
    end

    %% Crystal Shard Layer
    subgraph "Crystal Shard: tiktoken-cr"
        direction TB
        API["API Module"]:::crystal
        EncodeClass["Encoding Classes"]:::crystal
        FFI["FFI Binding"]:::crystal
    end

    %% Native Layer
    subgraph "Native Layer"
        direction TB
        DynLib["tiktoken-c Dynamic Library"]:::native
        Engine["Libtiktoken Engine"]:::native
    end

    %% Examples & Tests
    subgraph "Examples"
        direction TB
        Ex1["tanuki.cr"]:::examples
        Ex2["tanuki_special.cr"]:::examples
    end

    subgraph "Test Suite"
        direction TB
        Spec1["encoding_spec.cr"]:::tests
        Spec2["libtiktoken_spec.cr"]:::tests
        Spec3["tiktoken_spec.cr"]:::tests
    end

    %% CI/CD & Build Tools
    subgraph "CI/CD & Build Tools"
        direction TB
        Docs["docs.yml"]:::ci
        TestW["test.yml"]:::ci
        Make["Makefile"]:::ci
        ShardYml["shard.yml"]:::ci
    end

    %% Relationships
    App -->|"require + call encode/decode"| API
    API -->|"uses"| EncodeClass
    EncodeClass -->|"uses"| FFI
    FFI -->|"FFI (dlopen/link-flags)"| DynLib
    DynLib -->|"internal Rust functions"| Engine

    %% Examples & Tests interactions
    Ex1 -->|uses API| API
    Ex2 -->|uses API| API
    Spec1 -.->|tests Crystal API| API
    Spec2 -.->|tests FFI binding| FFI
    Spec3 -.->|tests shard integration| API

    %% CI/CD flows
    Docs -->|runs specs| Spec1
    Docs -->|runs specs| Spec2
    Docs -->|runs specs| Spec3
    TestW -->|runs cargo build| DynLib
    TestW -->|runs shard build| API
    Make -->|build script| DynLib
    ShardYml -->|shard config| API

    %% Click Events
    click API "https://github.com/kojix2/tiktoken-cr/blob/main/src/tiktoken.cr"
    click EncodeClass "https://github.com/kojix2/tiktoken-cr/blob/main/src/tiktoken/encoding.cr"
    click EncodeClass "https://github.com/kojix2/tiktoken-cr/blob/main/src/tiktoken/version.cr"
    click FFI "https://github.com/kojix2/tiktoken-cr/blob/main/src/tiktoken/libtiktoken.cr"
    click DynLib "https://github.com/kojix2/tiktoken-cr/tree/main/tiktoken-c/"
    click Spec1 "https://github.com/kojix2/tiktoken-cr/blob/main/spec/tiktoken/encoding_spec.cr"
    click Spec2 "https://github.com/kojix2/tiktoken-cr/blob/main/spec/tiktoken/libtiktoken_spec.cr"
    click Spec3 "https://github.com/kojix2/tiktoken-cr/blob/main/spec/tiktoken_spec.cr"
    click Ex1 "https://github.com/kojix2/tiktoken-cr/blob/main/examples/tanuki.cr"
    click Ex2 "https://github.com/kojix2/tiktoken-cr/blob/main/examples/tanuki_special.cr"
    click Docs "https://github.com/kojix2/tiktoken-cr/blob/main/.github/workflows/docs.yml"
    click TestW "https://github.com/kojix2/tiktoken-cr/blob/main/.github/workflows/test.yml"
    click Make "https://github.com/kojix2/tiktoken-cr/tree/main/Makefile"
    click ShardYml "https://github.com/kojix2/tiktoken-cr/blob/main/shard.yml"

    %% Styles
    classDef crystal fill:#B3CDE3,stroke:#333,stroke-width:1px
    classDef native fill:#CCEBC5,stroke:#333,stroke-width:1px
    classDef tests fill:#F2F2F2,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
    classDef ci fill:#E5E5E5,stroke:#333,stroke-width:1px
    classDef examples fill:#DECBE4,stroke:#333,stroke-width:1px
```