# thesumitbanik/tokentoken architecture

> Architecture diagram of the GitHub repository thesumitbanik/tokentoken, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thesumitbanik/tokentoken
- Repository: https://github.com/thesumitbanik/tokentoken

## Overview

tokentoken is a Python SDK and CLI for compressing text into dense LLM-readable form, interpreting compressed text, and measuring compression. The main workflow is input through the CLI or SDK, prompt selection and compression through a provider, then results and metrics. The SDK also exposes agent-memory, multi-agent, cross-model, and context-extension capabilities. Source excerpts show the core SDK, prompt, utility, and provider-routing relationships; additional capability links are based on the README, since some SDK and CLI sections were not fully sampled.

Main entry points: [`tokentoken/sdk.py`](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py) (Python SDK), [`tokentoken/router.py`](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/router.py) (Provider Router), [`tokentoken/cli.py`](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/cli.py) (CLI Commands), [`tokentoken/prompts.py`](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/prompts.py) (Compression Modes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_cli["CLI Commands<br/>[cli.py]"]
  node_sdk["Python SDK<br/>[sdk.py]"]
end

subgraph group_compression["Compression Core"]
  node_prompts["Compression Modes<br/>[prompts.py]"]
  node_utils["Token Utilities<br/>[utils.py]"]
  node_compression_result["Compression Results<br/>[sdk.py]"]
end

subgraph group_capabilities["SDK Capabilities"]
  node_decompression["Decompression<br/>[sdk.py]"]
  node_analytics["Compression Analytics<br/>[utils.py]"]
  node_agent_memory["Agent Memory<br/>[sdk.py]"]
  node_multi_agent["Multi-Agent Messaging<br/>[sdk.py]"]
  node_cross_model["Cross-Model Check<br/>[sdk.py]"]
  node_context_extension["Context Extension<br/>[sdk.py]"]
end

subgraph group_providers["Model Providers"]
  node_router["Provider Router<br/>[router.py]"]
  node_models{{"LLM Providers"}}
end

node_user(("User"))

node_user -->|"runs commands"| node_cli
node_user -->|"calls API"| node_sdk
node_cli -->|"invokes operations"| node_sdk
node_sdk -->|"selects prompts"| node_prompts
node_sdk -->|"counts and validates"| node_utils
node_sdk -->|"dispatches requests"| node_router
node_router -->|"calls provider"| node_models
node_sdk -->|"returns metrics"| node_compression_result
node_sdk -->|"interprets text"| node_decompression
node_sdk -->|"computes metrics"| node_analytics
node_sdk -->|"compresses sessions"| node_agent_memory
node_sdk -->|"compresses messages"| node_multi_agent
node_sdk -->|"checks readability"| node_cross_model
node_sdk -->|"compresses long text"| node_context_extension
node_agent_memory -->|"sends compression requests"| node_router
node_multi_agent -->|"sends compression requests"| node_router
node_cross_model -->|"queries reader model"| node_router
node_context_extension -->|"sends chunk requests"| node_router

click node_cli "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/cli.py"
click node_sdk "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py"
click node_prompts "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/prompts.py"
click node_utils "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/utils.py"
click node_router "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/router.py"
click node_compression_result "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py"
click node_decompression "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py"
click node_analytics "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/utils.py"
click node_agent_memory "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py"
click node_multi_agent "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py"
click node_cross_model "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py"
click node_context_extension "https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_sdk,node_user toneBlue
class node_prompts,node_utils,node_compression_result toneAmber
class node_decompression,node_analytics,node_agent_memory,node_multi_agent,node_cross_model,node_context_extension toneMint
class node_router,node_models toneRose
```

## Components

### User Interfaces

- [CLI Commands](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/cli.py): `tokentoken/cli.py`
- [Python SDK](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py): `tokentoken/sdk.py`

### Compression Core

- [Compression Modes](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/prompts.py): `tokentoken/prompts.py`
- [Token Utilities](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/utils.py): `tokentoken/utils.py`
- [Compression Results](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py): `tokentoken/sdk.py`

### SDK Capabilities

- [Decompression](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py): `tokentoken/sdk.py`
- [Compression Analytics](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/utils.py): `tokentoken/utils.py`
- [Agent Memory](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py): `tokentoken/sdk.py`
- [Multi-Agent Messaging](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py): `tokentoken/sdk.py`
- [Cross-Model Check](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py): `tokentoken/sdk.py`
- [Context Extension](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/sdk.py): `tokentoken/sdk.py`

### Model Providers

- [Provider Router](https://github.com/thesumitbanik/tokentoken/blob/main/tokentoken/router.py): `tokentoken/router.py`
- **LLM Providers**

### Other components

- **User**

## Connections

- User → CLI Commands: runs commands (evidence: `README.md`)
- User → Python SDK: calls API (evidence: `README.md`)
- CLI Commands → Python SDK: invokes operations (evidence: `tokentoken/cli.py`)
- Python SDK → Compression Modes: selects prompts (evidence: `tokentoken/sdk.py`)
- Python SDK → Token Utilities: counts and validates (evidence: `tokentoken/sdk.py`)
- Python SDK → Provider Router: dispatches requests (evidence: `tokentoken/sdk.py`)
- Provider Router → LLM Providers: calls provider (evidence: `tokentoken/router.py`)
- Python SDK → Compression Results: returns metrics (evidence: `tokentoken/sdk.py`)
- Python SDK → Decompression: interprets text (evidence: `README.md`)
- Python SDK → Compression Analytics: computes metrics (evidence: `tokentoken/sdk.py`)
- Python SDK → Agent Memory: compresses sessions (evidence: `tokentoken/sdk.py`)
- Python SDK → Multi-Agent Messaging: compresses messages (evidence: `tokentoken/sdk.py`)
- Python SDK → Cross-Model Check: checks readability (evidence: `tokentoken/sdk.py`)
- Python SDK → Context Extension: compresses long text (evidence: `tokentoken/sdk.py`)
- Agent Memory → Provider Router: sends compression requests (evidence: `tokentoken/sdk.py`)
- Multi-Agent Messaging → Provider Router: sends compression requests (evidence: `tokentoken/sdk.py`)
- Cross-Model Check → Provider Router: queries reader model (evidence: `tokentoken/sdk.py`)
- Context Extension → Provider Router: sends chunk requests (evidence: `tokentoken/sdk.py`)
