How najadsayshi/llm-gateway works
LLM Gateway provides an OpenAI-compatible API: callers submit chat requests with gateway keys, then the gateway authenticates and rate-limits them, routes by model to a configured provider, and returns a completion or stream. Non-streaming completions can be cached and are recorded for token and cost reporting; callers can query their usage separately. The graph includes key and usage persistence, Redis-backed limiting and caching, and provider integrations. Edges reflect shown calls or README-documented behavior; some storage and cache internals are represented at subsystem level.
Main entry points: chat.py Chat endpoint · main.py Application runtime · auth.py Key authentication · usage.py Usage accounting
Components 20
API workflow
- Chat endpoint
app/routers/chat.py - Request schema
app/schemas.py - Usage endpoint
app/routers/usage.py - Rate limiting
app/rate_limit.py - Application runtime
app/main.py
Provider routing
- Model registry
app/providers/registry.py - Provider client
app/providers/compatible.py - Retry policy
app/retry.py
Keys and usage
- Key authentication
app/auth.py - Key hashing
app/security.py - Usage accounting
app/usage.py - Cost calculation
app/pricing.py
Runtime state
- Response cache
app/cache.py - Persistence models
app/models.py - Database sessions
app/db.py - Settings
app/config.py
Other components
- OpenAI client
- LLM providers
- Postgres
- Redis
Connections 28
- OpenAI client → Chat endpoint: submits chat
readme.md - OpenAI client → Usage endpoint: requests totals
readme.md - Chat endpoint → Request schema: validates with
app/routers/chat.py - Chat endpoint → Rate limiting: depends on
app/routers/chat.py - Rate limiting → Key authentication: authenticates via
app/rate_limit.py - Key authentication → Key hashing: hashes key with
app/auth.py - Key authentication → Database sessions: queries through
app/auth.py - Key authentication → Persistence models: looks up key
app/auth.py - Rate limiting → Redis: increments counter
readme.md - Chat endpoint → Model registry: resolves model
app/routers/chat.py - Chat endpoint → Provider client: sends completion
app/routers/chat.py - Provider client → Retry policy: uses policy
app/providers/compatible.py - Provider client → LLM providers: sends request
readme.md - Chat endpoint → Response cache: gets and sets
app/routers/chat.py - Response cache → Redis: stores responses
readme.md - Chat endpoint → Usage accounting: records completion
app/routers/chat.py - Usage accounting → Cost calculation: calculates cost
app/usage.py - Usage accounting → Persistence models: creates and aggregates
app/usage.py - Usage accounting → Database sessions: uses session
app/usage.py - Usage endpoint → Key authentication: authenticates via
app/routers/usage.py - Usage endpoint → Usage accounting: gets summary
app/routers/usage.py - Database sessions → Postgres: connects to
readme.md - Persistence models → Database sessions: uses base
app/models.py - Application runtime → Settings: loads settings
app/main.py - Application runtime → Model registry: starts configured providers
app/main.py - Application runtime → Chat endpoint: registers route
app/main.py - Application runtime → Usage endpoint: registers route
app/main.py - Application runtime → Redis: opens connection
app/main.py