# najadsayshi/llm-gateway architecture

> Architecture diagram of the GitHub repository najadsayshi/llm-gateway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/najadsayshi/llm-gateway
- Repository: https://github.com/najadsayshi/llm-gateway

## Overview

LLM Gateway provides an OpenAI-compatible API: callers submit chat requests with gateway keys, then the gateway authenticates and rate-limits them, routes by model to a configured provider, and returns a completion or stream. Non-streaming completions can be cached and are recorded for token and cost reporting; callers can query their usage separately. The graph includes key and usage persistence, Redis-backed limiting and caching, and provider integrations. Edges reflect shown calls or README-documented behavior; some storage and cache internals are represented at subsystem level.

Main entry points: [`app/routers/chat.py`](https://github.com/najadsayshi/llm-gateway/blob/main/app/routers/chat.py) (Chat endpoint), [`app/main.py`](https://github.com/najadsayshi/llm-gateway/blob/main/app/main.py) (Application runtime), [`app/auth.py`](https://github.com/najadsayshi/llm-gateway/blob/main/app/auth.py) (Key authentication), [`app/usage.py`](https://github.com/najadsayshi/llm-gateway/blob/main/app/usage.py) (Usage accounting)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API workflow"]
  node_chat_api["Chat endpoint<br/>[chat.py]"]
  node_request_schema["Request schema<br/>[schemas.py]"]
  node_usage_api["Usage endpoint<br/>[usage.py]"]
  node_rate_limit["Rate limiting<br/>[rate_limit.py]"]
  node_runtime["Application runtime<br/>[main.py]"]
end

subgraph group_providers["Provider routing"]
  node_registry["Model registry<br/>[registry.py]"]
  node_compatible["Provider client<br/>[compatible.py]"]
  node_retry["Retry policy<br/>[retry.py]"]
end

subgraph group_accounting["Keys and usage"]
  node_auth["Key authentication<br/>[auth.py]"]
  node_security["Key hashing<br/>[security.py]"]
  node_usage_logic["Usage accounting<br/>[usage.py]"]
  node_pricing["Cost calculation<br/>[pricing.py]"]
end

subgraph group_state["Runtime state"]
  node_cache["Response cache<br/>[cache.py]"]
  node_models["Persistence models<br/>[models.py]"]
  node_db_session["Database sessions<br/>[db.py]"]
  node_config["Settings<br/>[config.py]"]
end

node_client(("OpenAI client"))
node_upstreams["LLM providers"]
node_database[("Postgres")]
node_redis[("Redis")]

node_client -->|"submits chat"| node_chat_api
node_client -->|"requests totals"| node_usage_api
node_chat_api -->|"validates with"| node_request_schema
node_chat_api -->|"depends on"| node_rate_limit
node_rate_limit -->|"authenticates via"| node_auth
node_auth -->|"hashes key with"| node_security
node_auth -->|"queries through"| node_db_session
node_auth -->|"looks up key"| node_models
node_rate_limit -->|"increments counter"| node_redis
node_chat_api -->|"resolves model"| node_registry
node_chat_api -->|"sends completion"| node_compatible
node_compatible -->|"uses policy"| node_retry
node_compatible -->|"sends request"| node_upstreams
node_chat_api -->|"gets and sets"| node_cache
node_cache -->|"stores responses"| node_redis
node_chat_api -->|"records completion"| node_usage_logic
node_usage_logic -->|"calculates cost"| node_pricing
node_usage_logic -->|"creates and aggregates"| node_models
node_usage_logic -->|"uses session"| node_db_session
node_usage_api -->|"authenticates via"| node_auth
node_usage_api -->|"gets summary"| node_usage_logic
node_db_session -->|"connects to"| node_database
node_models -->|"uses base"| node_db_session
node_runtime -->|"loads settings"| node_config
node_runtime -->|"starts configured providers"| node_registry
node_runtime -->|"registers route"| node_chat_api
node_runtime -->|"registers route"| node_usage_api
node_runtime -->|"opens connection"| node_redis

click node_chat_api "https://github.com/najadsayshi/llm-gateway/blob/main/app/routers/chat.py"
click node_request_schema "https://github.com/najadsayshi/llm-gateway/blob/main/app/schemas.py"
click node_usage_api "https://github.com/najadsayshi/llm-gateway/blob/main/app/routers/usage.py"
click node_auth "https://github.com/najadsayshi/llm-gateway/blob/main/app/auth.py"
click node_security "https://github.com/najadsayshi/llm-gateway/blob/main/app/security.py"
click node_rate_limit "https://github.com/najadsayshi/llm-gateway/blob/main/app/rate_limit.py"
click node_registry "https://github.com/najadsayshi/llm-gateway/blob/main/app/providers/registry.py"
click node_compatible "https://github.com/najadsayshi/llm-gateway/blob/main/app/providers/compatible.py"
click node_retry "https://github.com/najadsayshi/llm-gateway/blob/main/app/retry.py"
click node_cache "https://github.com/najadsayshi/llm-gateway/blob/main/app/cache.py"
click node_usage_logic "https://github.com/najadsayshi/llm-gateway/blob/main/app/usage.py"
click node_pricing "https://github.com/najadsayshi/llm-gateway/blob/main/app/pricing.py"
click node_models "https://github.com/najadsayshi/llm-gateway/blob/main/app/models.py"
click node_db_session "https://github.com/najadsayshi/llm-gateway/blob/main/app/db.py"
click node_config "https://github.com/najadsayshi/llm-gateway/blob/main/app/config.py"
click node_runtime "https://github.com/najadsayshi/llm-gateway/blob/main/app/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_chat_api,node_request_schema,node_usage_api,node_rate_limit,node_runtime,node_client toneBlue
class node_registry,node_compatible,node_retry,node_database,node_redis toneAmber
class node_auth,node_security,node_usage_logic,node_pricing toneMint
class node_cache,node_models,node_db_session,node_config toneRose
class node_upstreams toneIndigo
```

## Components

### API workflow

- [Chat endpoint](https://github.com/najadsayshi/llm-gateway/blob/main/app/routers/chat.py): `app/routers/chat.py`
- [Request schema](https://github.com/najadsayshi/llm-gateway/blob/main/app/schemas.py): `app/schemas.py`
- [Usage endpoint](https://github.com/najadsayshi/llm-gateway/blob/main/app/routers/usage.py): `app/routers/usage.py`
- [Rate limiting](https://github.com/najadsayshi/llm-gateway/blob/main/app/rate_limit.py): `app/rate_limit.py`
- [Application runtime](https://github.com/najadsayshi/llm-gateway/blob/main/app/main.py): `app/main.py`

### Provider routing

- [Model registry](https://github.com/najadsayshi/llm-gateway/blob/main/app/providers/registry.py): `app/providers/registry.py`
- [Provider client](https://github.com/najadsayshi/llm-gateway/blob/main/app/providers/compatible.py): `app/providers/compatible.py`
- [Retry policy](https://github.com/najadsayshi/llm-gateway/blob/main/app/retry.py): `app/retry.py`

### Keys and usage

- [Key authentication](https://github.com/najadsayshi/llm-gateway/blob/main/app/auth.py): `app/auth.py`
- [Key hashing](https://github.com/najadsayshi/llm-gateway/blob/main/app/security.py): `app/security.py`
- [Usage accounting](https://github.com/najadsayshi/llm-gateway/blob/main/app/usage.py): `app/usage.py`
- [Cost calculation](https://github.com/najadsayshi/llm-gateway/blob/main/app/pricing.py): `app/pricing.py`

### Runtime state

- [Response cache](https://github.com/najadsayshi/llm-gateway/blob/main/app/cache.py): `app/cache.py`
- [Persistence models](https://github.com/najadsayshi/llm-gateway/blob/main/app/models.py): `app/models.py`
- [Database sessions](https://github.com/najadsayshi/llm-gateway/blob/main/app/db.py): `app/db.py`
- [Settings](https://github.com/najadsayshi/llm-gateway/blob/main/app/config.py): `app/config.py`

### Other components

- **OpenAI client**
- **LLM providers**
- **Postgres**
- **Redis**

## Connections

- OpenAI client → Chat endpoint: submits chat (evidence: `readme.md`)
- OpenAI client → Usage endpoint: requests totals (evidence: `readme.md`)
- Chat endpoint → Request schema: validates with (evidence: `app/routers/chat.py`)
- Chat endpoint → Rate limiting: depends on (evidence: `app/routers/chat.py`)
- Rate limiting → Key authentication: authenticates via (evidence: `app/rate_limit.py`)
- Key authentication → Key hashing: hashes key with (evidence: `app/auth.py`)
- Key authentication → Database sessions: queries through (evidence: `app/auth.py`)
- Key authentication → Persistence models: looks up key (evidence: `app/auth.py`)
- Rate limiting → Redis: increments counter (evidence: `readme.md`)
- Chat endpoint → Model registry: resolves model (evidence: `app/routers/chat.py`)
- Chat endpoint → Provider client: sends completion (evidence: `app/routers/chat.py`)
- Provider client → Retry policy: uses policy (evidence: `app/providers/compatible.py`)
- Provider client → LLM providers: sends request (evidence: `readme.md`)
- Chat endpoint → Response cache: gets and sets (evidence: `app/routers/chat.py`)
- Response cache → Redis: stores responses (evidence: `readme.md`)
- Chat endpoint → Usage accounting: records completion (evidence: `app/routers/chat.py`)
- Usage accounting → Cost calculation: calculates cost (evidence: `app/usage.py`)
- Usage accounting → Persistence models: creates and aggregates (evidence: `app/usage.py`)
- Usage accounting → Database sessions: uses session (evidence: `app/usage.py`)
- Usage endpoint → Key authentication: authenticates via (evidence: `app/routers/usage.py`)
- Usage endpoint → Usage accounting: gets summary (evidence: `app/routers/usage.py`)
- Database sessions → Postgres: connects to (evidence: `readme.md`)
- Persistence models → Database sessions: uses base (evidence: `app/models.py`)
- Application runtime → Settings: loads settings (evidence: `app/main.py`)
- Application runtime → Model registry: starts configured providers (evidence: `app/main.py`)
- Application runtime → Chat endpoint: registers route (evidence: `app/main.py`)
- Application runtime → Usage endpoint: registers route (evidence: `app/main.py`)
- Application runtime → Redis: opens connection (evidence: `app/main.py`)
