How mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing works
Tollgate is a multi-tenant, OpenAI-compatible LLM gateway with a tenant analytics dashboard. Its central workflow accepts an authenticated chat request, applies rate and budget controls, checks exact and semantic caches, routes to an upstream model with retry/failover, then publishes usage for durable accounting. The dashboard exposes usage, cost, budgets, provider health, caching, routing, and request detail. The README documents these capabilities; sampled excerpts do not show every runtime wiring path, so unsupported connections are omitted or marked unverified.
Main entry points: chat.py Chat API · service.py Exact Cache · client.ts Dashboard API Client · App.tsx Dashboard App
Components 22
Gateway Workflow
- Chat API
apps/gateway/src/api/routes/chat.py - Authentication
apps/gateway/src/api/routes/auth.py - Provider Adapters
apps/gateway/src/providers/registry.py
Cost and Reliability
- Rate Limiting
apps/gateway/src/ratelimit/limiter.py - Budget Reservation
apps/gateway/src/budgets/manager.py - Exact Cache
apps/gateway/src/cache/service.py - Semantic Cache
apps/gateway/src/cache/semantic/service.py - Model Router
apps/gateway/src/router/service.py - Retry and Failover
apps/gateway/src/reliability/executor.py
Usage Accounting
- Usage Publisher
apps/gateway/src/usage/publisher.py - Usage Event Stream
- Usage Worker
apps/worker/src/consumer.py - Usage Persistence
apps/worker/src/persistence.py
Tenant Dashboard
- Dashboard App
apps/dashboard/src/App.tsx - Dashboard API Client
apps/dashboard/src/api/client.ts - Analytics Pages
apps/dashboard/src/pages/RouterPage.tsx - Dashboard Analytics
apps/gateway/src/services/dashboard_service.py
Shared Platform
- Observability
packages/core/src/tollgate_core/observability/metrics.py
Other components
- API Client
- LLM Providers
- PostgreSQL
- Redis
Connections 24
- API Client → Chat API: submits request
README.md - API Client → Authentication: authenticates
README.md - Chat API → Rate Limiting: enforces limits
README.md - Chat API → Budget Reservation: reserves spend
README.md - Chat API → Exact Cache: checks exact
README.md - Chat API → Semantic Cache: checks semantic
README.md - Chat API → Model Router: selects model
README.md - Chat API → Provider Adapters: dispatches request
README.md - Provider Adapters → Retry and Failover: uses retries
README.md - Retry and Failover → LLM Providers: calls provider
README.md - Chat API → Usage Publisher: publishes usage
README.md - Usage Publisher → Usage Event Stream: appends events
README.md - Usage Event Stream → Usage Worker: delivers events
README.md - Usage Worker → Usage Persistence: persists records
apps/worker/src/consumer.py - Usage Persistence → PostgreSQL: writes records
apps/worker/src/persistence.py - Budget Reservation → Redis: reserves atomically
apps/gateway/src/budgets/manager.py - Exact Cache → Redis: reads and writes
apps/gateway/src/cache/service.py - Semantic Cache → Exact Cache: fetches response
apps/gateway/src/cache/semantic/service.py - Dashboard App → Dashboard API Client: requests data
apps/dashboard/src/App.tsx - Dashboard App → Analytics Pages: renders views
apps/dashboard/src/App.tsx - Analytics Pages → Dashboard API Client: loads analytics
apps/dashboard/src/pages/RouterPage.tsx - Dashboard API Client → Dashboard Analytics: requests analytics
- Dashboard Analytics → PostgreSQL: queries analytics
- Chat API → Observability: records telemetry