# mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing architecture

> Architecture diagram of the GitHub repository mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing
- Repository: https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing

## Overview

Tollgate is a multi-tenant, OpenAI-compatible LLM gateway with a tenant analytics dashboard. Its central workflow accepts an authenticated chat request, applies rate and budget controls, checks exact and semantic caches, routes to an upstream model with retry/failover, then publishes usage for durable accounting. The dashboard exposes usage, cost, budgets, provider health, caching, routing, and request detail. The README documents these capabilities; sampled excerpts do not show every runtime wiring path, so unsupported connections are omitted or marked unverified.

Main entry points: [`apps/gateway/src/api/routes/chat.py`](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/api/routes/chat.py) (Chat API), [`apps/gateway/src/cache/service.py`](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/cache/service.py) (Exact Cache), [`apps/dashboard/src/api/client.ts`](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/api/client.ts) (Dashboard API Client), [`apps/dashboard/src/App.tsx`](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/App.tsx) (Dashboard App)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_gateway["Gateway Workflow"]
  node_chat_api["Chat API<br/>[chat.py]"]
  node_auth["Authentication<br/>[auth.py]"]
  node_providers["Provider Adapters<br/>[registry.py]"]
end

subgraph group_controls["Cost and Reliability"]
  node_ratelimit["Rate Limiting<br/>[limiter.py]"]
  node_budgets["Budget Reservation<br/>[manager.py]"]
  node_exact_cache["Exact Cache<br/>[service.py]"]
  node_semantic_cache["Semantic Cache<br/>[service.py]"]
  node_router["Model Router<br/>[service.py]"]
  node_reliability["Retry and Failover<br/>[executor.py]"]
end

subgraph group_accounting["Usage Accounting"]
  node_usage_publisher["Usage Publisher<br/>[publisher.py]"]
  node_usage_stream["Usage Event Stream"]
  node_worker["Usage Worker<br/>[consumer.py]"]
  node_persistence["Usage Persistence<br/>[persistence.py]"]
end

subgraph group_dashboard["Tenant Dashboard"]
  node_dashboard_app["Dashboard App<br/>[App.tsx]"]
  node_dashboard_api["Dashboard API Client<br/>[client.ts]"]
  node_dashboard_pages["Analytics Pages<br/>[RouterPage.tsx]"]
  node_dashboard_service["Dashboard Analytics"]
end

subgraph group_shared["Shared Platform"]
  node_observability["Observability<br/>[metrics.py]"]
end

node_client(("API Client"))
node_provider_service["LLM Providers"]
node_database[("PostgreSQL")]
node_redis[("Redis")]

node_client -->|"submits request"| node_chat_api
node_client -->|"authenticates"| node_auth
node_chat_api -->|"enforces limits"| node_ratelimit
node_chat_api -->|"reserves spend"| node_budgets
node_chat_api -->|"checks exact"| node_exact_cache
node_chat_api -->|"checks semantic"| node_semantic_cache
node_chat_api -->|"selects model"| node_router
node_chat_api -->|"dispatches request"| node_providers
node_providers -->|"uses retries"| node_reliability
node_reliability -->|"calls provider"| node_provider_service
node_chat_api -->|"publishes usage"| node_usage_publisher
node_usage_publisher -->|"appends events"| node_usage_stream
node_usage_stream -->|"delivers events"| node_worker
node_worker -->|"persists records"| node_persistence
node_persistence -->|"writes records"| node_database
node_budgets -->|"reserves atomically"| node_redis
node_exact_cache -->|"reads and writes"| node_redis
node_semantic_cache -->|"fetches response"| node_exact_cache
node_dashboard_app -->|"requests data"| node_dashboard_api
node_dashboard_app -->|"renders views"| node_dashboard_pages
node_dashboard_pages -->|"loads analytics"| node_dashboard_api
node_dashboard_api -.->|"requests analytics"| node_dashboard_service
node_dashboard_service -.->|"queries analytics"| node_database
node_chat_api -->|"records telemetry"| node_observability

click node_chat_api "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/api/routes/chat.py"
click node_auth "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/api/routes/auth.py"
click node_ratelimit "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/ratelimit/limiter.py"
click node_budgets "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/budgets/manager.py"
click node_exact_cache "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/cache/service.py"
click node_semantic_cache "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/cache/semantic/service.py"
click node_router "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/router/service.py"
click node_providers "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/providers/registry.py"
click node_reliability "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/reliability/executor.py"
click node_usage_publisher "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/usage/publisher.py"
click node_worker "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/worker/src/consumer.py"
click node_persistence "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/worker/src/persistence.py"
click node_dashboard_app "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/App.tsx"
click node_dashboard_api "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/api/client.ts"
click node_dashboard_pages "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/pages/RouterPage.tsx"
click node_dashboard_service "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/services/dashboard_service.py"
click node_observability "https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/packages/core/src/tollgate_core/observability/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chat_api,node_auth,node_providers,node_client toneBlue
class node_ratelimit,node_budgets,node_exact_cache,node_semantic_cache,node_router,node_reliability,node_database,node_redis toneAmber
class node_usage_publisher,node_usage_stream,node_worker,node_persistence toneMint
class node_dashboard_app,node_dashboard_api,node_dashboard_pages,node_dashboard_service toneRose
class node_observability,node_provider_service toneIndigo
```

## Components

### Gateway Workflow

- [Chat API](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/api/routes/chat.py): `apps/gateway/src/api/routes/chat.py`
- [Authentication](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/api/routes/auth.py): `apps/gateway/src/api/routes/auth.py`
- [Provider Adapters](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/providers/registry.py): `apps/gateway/src/providers/registry.py`

### Cost and Reliability

- [Rate Limiting](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/ratelimit/limiter.py): `apps/gateway/src/ratelimit/limiter.py`
- [Budget Reservation](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/budgets/manager.py): `apps/gateway/src/budgets/manager.py`
- [Exact Cache](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/cache/service.py): `apps/gateway/src/cache/service.py`
- [Semantic Cache](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/cache/semantic/service.py): `apps/gateway/src/cache/semantic/service.py`
- [Model Router](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/router/service.py): `apps/gateway/src/router/service.py`
- [Retry and Failover](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/reliability/executor.py): `apps/gateway/src/reliability/executor.py`

### Usage Accounting

- [Usage Publisher](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/usage/publisher.py): `apps/gateway/src/usage/publisher.py`
- **Usage Event Stream**
- [Usage Worker](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/worker/src/consumer.py): `apps/worker/src/consumer.py`
- [Usage Persistence](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/worker/src/persistence.py): `apps/worker/src/persistence.py`

### Tenant Dashboard

- [Dashboard App](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/App.tsx): `apps/dashboard/src/App.tsx`
- [Dashboard API Client](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/api/client.ts): `apps/dashboard/src/api/client.ts`
- [Analytics Pages](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/dashboard/src/pages/RouterPage.tsx): `apps/dashboard/src/pages/RouterPage.tsx`
- [Dashboard Analytics](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/apps/gateway/src/services/dashboard_service.py): `apps/gateway/src/services/dashboard_service.py`

### Shared Platform

- [Observability](https://github.com/mayanksingh2745/tollgate-a-multi-tenant-llm-gateway-with-budgets-semantic-caching-and-cost-quality-routing/blob/main/packages/core/src/tollgate_core/observability/metrics.py): `packages/core/src/tollgate_core/observability/metrics.py`

### Other components

- **API Client**
- **LLM Providers**
- **PostgreSQL**
- **Redis**

## Connections

- API Client → Chat API: submits request (evidence: `README.md`)
- API Client → Authentication: authenticates (evidence: `README.md`)
- Chat API → Rate Limiting: enforces limits (evidence: `README.md`)
- Chat API → Budget Reservation: reserves spend (evidence: `README.md`)
- Chat API → Exact Cache: checks exact (evidence: `README.md`)
- Chat API → Semantic Cache: checks semantic (evidence: `README.md`)
- Chat API → Model Router: selects model (evidence: `README.md`)
- Chat API → Provider Adapters: dispatches request (evidence: `README.md`)
- Provider Adapters → Retry and Failover: uses retries (evidence: `README.md`)
- Retry and Failover → LLM Providers: calls provider (evidence: `README.md`)
- Chat API → Usage Publisher: publishes usage (evidence: `README.md`)
- Usage Publisher → Usage Event Stream: appends events (evidence: `README.md`)
- Usage Event Stream → Usage Worker: delivers events (evidence: `README.md`)
- Usage Worker → Usage Persistence: persists records (evidence: `apps/worker/src/consumer.py`)
- Usage Persistence → PostgreSQL: writes records (evidence: `apps/worker/src/persistence.py`)
- Budget Reservation → Redis: reserves atomically (evidence: `apps/gateway/src/budgets/manager.py`)
- Exact Cache → Redis: reads and writes (evidence: `apps/gateway/src/cache/service.py`)
- Semantic Cache → Exact Cache: fetches response (evidence: `apps/gateway/src/cache/semantic/service.py`)
- Dashboard App → Dashboard API Client: requests data (evidence: `apps/dashboard/src/App.tsx`)
- Dashboard App → Analytics Pages: renders views (evidence: `apps/dashboard/src/App.tsx`)
- Analytics Pages → Dashboard API Client: loads analytics (evidence: `apps/dashboard/src/pages/RouterPage.tsx`)
- Dashboard API Client → Dashboard Analytics: requests analytics
- Dashboard Analytics → PostgreSQL: queries analytics
- Chat API → Observability: records telemetry
