How kojix2/tiktoken-cr works
Project type and purpose
• From the README and file tree, this is not a web-app or full-stack system but a language-binding library (“shard”) for Crystal.
• It wraps/links to an external Rust/C implementation (“tiktoken-c”) to provide GPT tokenization to Crystal programs.
File structure
• Top-level:
• shard.yml, Makefile, README.md, LICENSE, .gitignore, .github/ (CI/workflows)
• src/ – Crystal wrapper code (tiktoken.cr plus a tiktoken/ subfolder)
• tiktoken-c/ – Git submodule containing the Rust source built as a dynamic library via Cargo
• spec/ – Crystal unit tests against both the pure-Crystal stubs and the underlying C library
• examples/ – sample scripts showing how end users call the API
• Key patterns:
• FFI boundary: Crystal code in src/tiktoken/libtiktoken.cr calls into the compiled tiktoken-c dynamic library.
• “Core” logic sits in the Rust crate; Crystal provides idiomatic API & type definitions.
README for additional insights
• Installation steps show build flow:
Crystal program → requires “tiktoken” shard → at compile time links to tiktoken-c via LD_LIBRARY_PATH or --link-flags.
• Usage snippets reveal the two API layers:
• Tiktoken.encoding_for_model / Tiktoken::Encoding.* as the Crystal interface
• encode/decode methods that trigger Rust library functionality