How thesumitbanik/tokentoken works
tokentoken is a Python SDK and CLI for compressing text into dense LLM-readable form, interpreting compressed text, and measuring compression. The main workflow is input through the CLI or SDK, prompt selection and compression through a provider, then results and metrics. The SDK also exposes agent-memory, multi-agent, cross-model, and context-extension capabilities. Source excerpts show the core SDK, prompt, utility, and provider-routing relationships; additional capability links are based on the README, since some SDK and CLI sections were not fully sampled.
Main entry points: sdk.py Python SDK · router.py Provider Router · cli.py CLI Commands · prompts.py Compression Modes
Components 14
User Interfaces
- CLI Commands
tokentoken/cli.py - Python SDK
tokentoken/sdk.py
Compression Core
- Compression Modes
tokentoken/prompts.py - Token Utilities
tokentoken/utils.py - Compression Results
tokentoken/sdk.py
SDK Capabilities
- Decompression
tokentoken/sdk.py - Compression Analytics
tokentoken/utils.py - Agent Memory
tokentoken/sdk.py - Multi-Agent Messaging
tokentoken/sdk.py - Cross-Model Check
tokentoken/sdk.py - Context Extension
tokentoken/sdk.py
Model Providers
- Provider Router
tokentoken/router.py - LLM Providers
Other components
- User
Connections 18
- User → CLI Commands: runs commands
README.md - User → Python SDK: calls API
README.md - CLI Commands → Python SDK: invokes operations
tokentoken/cli.py - Python SDK → Compression Modes: selects prompts
tokentoken/sdk.py - Python SDK → Token Utilities: counts and validates
tokentoken/sdk.py - Python SDK → Provider Router: dispatches requests
tokentoken/sdk.py - Provider Router → LLM Providers: calls provider
tokentoken/router.py - Python SDK → Compression Results: returns metrics
tokentoken/sdk.py - Python SDK → Decompression: interprets text
README.md - Python SDK → Compression Analytics: computes metrics
tokentoken/sdk.py - Python SDK → Agent Memory: compresses sessions
tokentoken/sdk.py - Python SDK → Multi-Agent Messaging: compresses messages
tokentoken/sdk.py - Python SDK → Cross-Model Check: checks readability
tokentoken/sdk.py - Python SDK → Context Extension: compresses long text
tokentoken/sdk.py - Agent Memory → Provider Router: sends compression requests
tokentoken/sdk.py - Multi-Agent Messaging → Provider Router: sends compression requests
tokentoken/sdk.py - Cross-Model Check → Provider Router: queries reader model
tokentoken/sdk.py - Context Extension → Provider Router: sends chunk requests
tokentoken/sdk.py