How lilakhiz/tokenlean works
TokenLean is an LLM token-optimisation proxy: applications send compatible requests through it, the proxy applies configured optimisation and safety stages, then routes requests to an LLM provider and reports usage and savings. Its principal runtime is the proxy middleware pipeline; separate document and fine-tuning pipelines support knowledge preparation. The README documents the main capabilities and provider breadth; some subsystem links below reflect those documented roles because their source was not sampled.
Main entry points: pipeline.py Middleware pipeline · main.py Proxy API · g18_observability.py Observability · __init__.py Provider adapters
Components 25
Proxy runtime
- Proxy API
src/proxy/main.py - Middleware pipeline
src/proxy/middleware/pipeline.py - Tenant context
src/proxy/tenancy/context.py - Configuration
src/proxy/config_loader.py - API key management
src/proxy/auth/api_key_manager.py - Provider adapters
src/proxy/providers/__init__.py
Request optimisation
- Prompt compression
src/proxy/middleware/g01_compression.py - Prompt templates
src/proxy/middleware/g02_template_registry.py - Semantic cache
src/proxy/middleware/g05_cache.py - Model routing
src/proxy/middleware/g06_routing.py - Tool loading
src/proxy/middleware/g08_tool_loading.py - Output formatting
src/proxy/middleware/g11_output_format.py - Tool output reduction
src/proxy/middleware/g14_tool_output.py - Server-side compute
src/proxy/middleware/g15_server_compute.py
Knowledge pipelines
- Document trigger
src/proxy/middleware/g03_doc_pipeline.py - Retrieval
src/proxy/middleware/g07_retrieval.py - Document processing
src/doc-pipeline/pipeline.py - Fine-tuning pipeline
src/finetune-pipeline/pipeline.py
Safety and operations
- Guardrails
src/proxy/middleware/g30_guardrails.py - PII redaction
src/proxy/middleware/g29_pii_redaction.py - Usage metering
src/proxy/billing/metering.py - Observability
src/proxy/middleware/g18_observability.py - Savings calculation
src/proxy/savings/calculator.py
Other components
- Client application
- LLM providers
Connections 23
- Client application → Proxy API: sends request
- Proxy API → API key management: validates keys
src/proxy/main.py - Proxy API → Middleware pipeline: passes request
src/proxy/main.py - Middleware pipeline → Tenant context: resolves tenant
src/proxy/middleware/pipeline.py - Middleware pipeline → Configuration: loads settings
src/proxy/middleware/pipeline.py - Middleware pipeline → Prompt compression: optimises prompt
src/proxy/middleware/pipeline.py - Middleware pipeline → Prompt templates: applies templates
src/proxy/middleware/pipeline.py - Middleware pipeline → Semantic cache: checks cache
src/proxy/middleware/pipeline.py - Middleware pipeline → Model routing: routes model
src/proxy/middleware/pipeline.py - Middleware pipeline → Retrieval: adds context
src/proxy/middleware/pipeline.py - Middleware pipeline → Tool loading: prunes tools
src/proxy/middleware/pipeline.py - Middleware pipeline → Output formatting: formats output
src/proxy/middleware/pipeline.py - Middleware pipeline → Tool output reduction: reduces tool output
src/proxy/middleware/pipeline.py - Middleware pipeline → Server-side compute: offloads compute
src/proxy/middleware/pipeline.py - Middleware pipeline → Guardrails: checks safety
src/proxy/middleware/pipeline.py - Middleware pipeline → PII redaction: redacts PII
src/proxy/middleware/pipeline.py - Middleware pipeline → Provider adapters: dispatches request
src/proxy/middleware/pipeline.py - Provider adapters → LLM providers: calls provider
- Middleware pipeline → Observability: records usage
src/proxy/middleware/pipeline.py - Observability → Savings calculation: estimates savings
src/proxy/middleware/g18_observability.py - Observability → Usage metering: meters usage
- Document trigger → Document processing: triggers processing
src/proxy/middleware/g03_doc_pipeline.py - Document processing → Fine-tuning pipeline: feeds fine-tuning