# theopenco/llmgateway architecture

> Architecture diagram of the GitHub repository theopenco/llmgateway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/theopenco/llmgateway
- Repository: https://github.com/theopenco/llmgateway

## Overview

1. This repository is a multi-app monorepo for an LLM gateway product, split into a hot-path inference layer and a control-plane/product layer. The main runtime pieces are a Hono-based backend API, a dedicated gateway service, several Next.js apps, a background worker, and shared packages for data, models, caching, and utilities.

2. The core data-plane is `apps/gateway`. It accepts OpenAI-compatible requests and also exposes specialized endpoints for embeddings, images, OCR, speech, video, and responses. Its job is to normalize requests, choose providers and fallback paths, translate provider-specific formats, and enforce routing, rate limits, and capability checks before sending traffic to external LLM providers.

3. The gateway’s outbound dependencies are the actual model providers: OpenAI, Anthropic, Google/Vertex, AWS Bedrock, Azure, DeepSeek, Mistral, xAI, etc. Provider/model metadata lives in `packages/models`, while request transformation and provider selection logic is concentrated in `packages/actions` and `apps/gateway/src/chat/tools`. That is the main abstraction boundary for “provider-agnostic” LLM access.

4. Persistence and operational state are centered on PostgreSQL and Redis. `packages/db` contains the schema, migrations, seed data, and query helpers for organizations, projects, API keys, logs, metrics, billing, discounts, and webhooks. Redis is used for caching, SWR-style query support, rate limiting, and transient gateway state. In diagrams, this is the shared state layer beneath the API, gateway, worker, and admin surfaces.

5. `apps/api` is the main backend control plane. It exposes endpoints for authentication, organizations, projects, keys, billing, analytics, audit logs, referrals, platform sessions, webhooks, and public configuration. This service is where the product’s account management and platform administration logic lives, distinct from the inference traffic handled by the gateway.

6. `apps/ui`, `apps/playground`, `apps/code`, and `ee/admin` are separate Next.js frontends with different audiences. `apps/ui` is the public marketing site plus the customer dashboard; `apps/playground` is the interactive product demo and authenticated sandbox; `apps/code` focuses on coding-agent/dev-plan content and user-facing plan management; `ee/admin` is an internal enterprise operations console. They all consume the backend API and shared data layer, but each has a distinct UX and domain scope.

7. `apps/worker` handles asynchronous and scheduled processing. It aggregates stats, syncs models, processes follow-up emails, and manages video-related jobs. This is the repository’s background-processing boundary: anything that should not block request/response flows is pushed here, usually reading from or writing to the same database and telemetry streams used by the API and gateway.

8. There is a clear separation between synchronous request handling and analytics/metadata processing. The gateway records logs and metrics about model usage, cost, latency, retries, and failures; the worker compacts or aggregates that data into project/global stats; the UI and admin dashboards query those aggregates for charts, tables, and cost/usage views. This is the main analytics data flow in the system.

9. Authentication and authorization are shared concerns but applied differently across surfaces. The API and dashboards use app-level auth, while the gateway also validates API tokens and provider credentials for inference requests. Enterprise and platform features such as org/team access, master keys, provider keys, billing, and guardrails sit behind these auth boundaries.

10. The `ee/` directory adds enterprise-only modules and UI. The notable runtime pieces are `ee/guardrails` for policy and redaction enforcement, `ee/audit` for structured audit logging, and `ee/admin` for the internal admin dashboard. These are layered on top of the core OSS platform and share the same general data and identity model.

11. Docs and content are first-class but architecturally separate. `apps/docs` is a documentation site with MDX content and API/search/proxy routes, while `apps/ui` and `apps/code` also host large content libraries for blog posts, changelogs, guides, comparisons, and legal pages. These sites are mostly read-only content delivery, not part of the runtime LLM request path.

12. Deployment supports both split and unified topologies. The repo includes Docker Compose variants, unified and split Dockerfiles, Helm charts, and run scripts. In a diagram, the unified deployment bundles the gateway, API, frontends, worker, PostgreSQL, and Redis; the split deployment treats the gateway as a separate service from the control-plane apps.

13. Shared libraries are intentionally kept small and domain-focused. `packages/shared` contains cross-app utilities like routing config, token estimation, load balancing, URL safety, fees, video access, and shared UI primitives; `packages/cache` and `packages/logger` provide infrastructure helpers; `packages/instrumentation` provides tracing, metrics, and request lifecycle hooks. These packages are the glue that keeps the monorepo consistent across services.

Main entry points: [`apps/api`](https://github.com/theopenco/llmgateway/tree/main/apps/api) (API), [`infra`](https://github.com/theopenco/llmgateway/tree/main/infra) (Deploy), [`apps/gateway/src/chat`](https://github.com/theopenco/llmgateway/tree/main/apps/gateway/src/chat) (Chat flow), [`apps/gateway`](https://github.com/theopenco/llmgateway/tree/main/apps/gateway) (Gateway)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data_plane["Data plane"]
  node_apps_gateway["Gateway"]
  node_gateway_chat["Chat flow<br/>gateway pipeline"]
  node_gateway_routes["Gateway APIs<br/>endpoint layer"]
end

subgraph group_control_plane["Control plane"]
  node_apps_api["API<br/>control-plane backend"]
  node_apps_ui["UI<br/>dashboard site"]
  node_apps_playground["Playground<br/>demo sandbox"]
  node_apps_code["Code site<br/>coding-agent site"]
  node_apps_docs["Docs<br/>documentation site"]
  node_apps_worker["Worker<br/>background jobs"]
end

subgraph group_shared_layer["Shared layer"]
  node_packages_actions["Actions<br/>request transforms"]
  node_packages_models["Models<br/>model catalog"]
  node_packages_db[("Database<br/>postgres schema")]
  node_packages_cache["Cache"]
  node_packages_shared["Shared utils<br/>cross-app helpers"]
  node_packages_instrumentation{{"Instrumentation<br/>telemetry"}}
end

subgraph group_enterprise["Enterprise"]
  node_ee_guardrails{{"Guardrails<br/>policy engine"}}
  node_ee_audit["Audit"]
  node_ee_admin["Admin<br/>enterprise console"]
end

subgraph group_ops["Operations"]
  node_infra_deploy{{"Deploy<br/>runtime topology"}}
end

node_apps_gateway -->|"serves"| node_gateway_chat
node_apps_gateway -->|"exposes"| node_gateway_routes
node_gateway_chat -->|"transforms via"| node_packages_actions
node_gateway_chat -->|"resolves"| node_packages_models
node_gateway_chat -->|"reads/writes"| node_packages_db
node_gateway_chat -->|"caches"| node_packages_cache
node_gateway_chat -->|"emits"| node_packages_instrumentation
node_gateway_routes -->|"uses"| node_packages_actions
node_gateway_routes -->|"validates"| node_packages_models
node_apps_gateway -->|"enforces"| node_ee_guardrails
node_apps_gateway -->|"logs"| node_ee_audit
node_apps_api -->|"persists"| node_packages_db
node_apps_api -->|"uses"| node_packages_cache
node_apps_api -->|"traces"| node_packages_instrumentation
node_apps_ui -->|"consumes"| node_apps_api
node_apps_playground -->|"consumes"| node_apps_api
node_apps_code -->|"consumes"| node_apps_api
node_apps_docs -->|"documents"| node_apps_api
node_apps_worker -->|"aggregates"| node_packages_db
node_apps_worker -->|"coordinates"| node_packages_cache
node_apps_worker -->|"reports"| node_packages_instrumentation
node_apps_worker -->|"supports"| node_apps_api
node_ee_admin -->|"administers"| node_apps_api
node_ee_admin -->|"queries"| node_packages_db
node_ee_admin -->|"reviews"| node_ee_audit
node_ee_guardrails -->|"guards"| node_apps_gateway
node_ee_guardrails -->|"configures"| node_apps_api
node_infra_deploy -->|"deploys"| node_apps_gateway
node_infra_deploy -->|"deploys"| node_apps_api
node_infra_deploy -->|"deploys"| node_apps_ui
node_infra_deploy -->|"deploys"| node_apps_playground
node_infra_deploy -->|"deploys"| node_apps_code
node_infra_deploy -->|"deploys"| node_apps_docs
node_infra_deploy -->|"deploys"| node_apps_worker
node_infra_deploy -->|"provisions"| node_packages_db
node_infra_deploy -->|"provisions"| node_packages_cache

click node_apps_gateway "https://github.com/theopenco/llmgateway/tree/main/apps/gateway"
click node_gateway_chat "https://github.com/theopenco/llmgateway/tree/main/apps/gateway/src/chat"
click node_gateway_routes "https://github.com/theopenco/llmgateway/tree/main/apps/gateway/src"
click node_packages_actions "https://github.com/theopenco/llmgateway/tree/main/packages/actions"
click node_packages_models "https://github.com/theopenco/llmgateway/tree/main/packages/models"
click node_packages_db "https://github.com/theopenco/llmgateway/tree/main/packages/db"
click node_packages_cache "https://github.com/theopenco/llmgateway/tree/main/packages/cache"
click node_packages_shared "https://github.com/theopenco/llmgateway/tree/main/packages/shared"
click node_packages_instrumentation "https://github.com/theopenco/llmgateway/tree/main/packages/instrumentation"
click node_apps_api "https://github.com/theopenco/llmgateway/tree/main/apps/api"
click node_apps_ui "https://github.com/theopenco/llmgateway/tree/main/apps/ui"
click node_apps_playground "https://github.com/theopenco/llmgateway/tree/main/apps/playground"
click node_apps_code "https://github.com/theopenco/llmgateway/tree/main/apps/code"
click node_apps_docs "https://github.com/theopenco/llmgateway/tree/main/apps/docs"
click node_apps_worker "https://github.com/theopenco/llmgateway/tree/main/apps/worker"
click node_ee_guardrails "https://github.com/theopenco/llmgateway/tree/main/ee/guardrails"
click node_ee_audit "https://github.com/theopenco/llmgateway/tree/main/ee/audit"
click node_ee_admin "https://github.com/theopenco/llmgateway/tree/main/ee/admin"
click node_infra_deploy "https://github.com/theopenco/llmgateway/tree/main/infra"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_apps_gateway,node_gateway_chat,node_gateway_routes toneBlue
class node_apps_api,node_apps_ui,node_apps_playground,node_apps_code,node_apps_docs,node_apps_worker toneAmber
class node_packages_actions,node_packages_models,node_packages_db,node_packages_cache,node_packages_shared,node_packages_instrumentation toneMint
class node_ee_guardrails,node_ee_audit,node_ee_admin toneRose
class node_infra_deploy toneIndigo
```

## Components

### Data plane

Hot-path inference and provider routing

- [Gateway](https://github.com/theopenco/llmgateway/tree/main/apps/gateway): inference gateway · OpenAI-compatible request handling, specialized multimodal endpoints, routing, fallback, and provider translation · `apps/gateway`
- [Chat flow](https://github.com/theopenco/llmgateway/tree/main/apps/gateway/src/chat): gateway pipeline · Normalizes chat requests, applies routing and fallback, and streams responses · `apps/gateway/src/chat`
- [Gateway APIs](https://github.com/theopenco/llmgateway/tree/main/apps/gateway/src): endpoint layer · Embeddings, images, OCR, speech, video, moderations, models, MCP, and responses endpoints · `apps/gateway/src`

### Control plane

Product APIs, dashboards, docs, and admin surfaces

- [API](https://github.com/theopenco/llmgateway/tree/main/apps/api): control-plane backend · Authentication, org/project management, keys, billing, analytics, audit logs, referrals, sessions, webhooks, and public config · `apps/api`
- [UI](https://github.com/theopenco/llmgateway/tree/main/apps/ui): dashboard site · Marketing site and customer dashboard for usage, billing, settings, and product content · `apps/ui`
- [Playground](https://github.com/theopenco/llmgateway/tree/main/apps/playground): demo sandbox · Interactive product demo for chat, image, audio, video, and org/project experiences · `apps/playground`
- [Code site](https://github.com/theopenco/llmgateway/tree/main/apps/code): coding-agent site · Coding-agent content, comparisons, and plan management for the dev audience · `apps/code`
- [Docs](https://github.com/theopenco/llmgateway/tree/main/apps/docs): documentation site · MDX docs, API/search/proxy routes, self-hosting guides, and feature docs · `apps/docs`
- [Worker](https://github.com/theopenco/llmgateway/tree/main/apps/worker): background jobs · Async aggregation, model sync, follow-up emails, and video jobs · `apps/worker`

### Shared layer

Common state, models, utilities, and telemetry

- [Actions](https://github.com/theopenco/llmgateway/tree/main/packages/actions): request transforms · Provider selection, request preparation, message transforms, headers, and endpoint resolution · `packages/actions`
- [Models](https://github.com/theopenco/llmgateway/tree/main/packages/models): model catalog · Provider and model metadata, capabilities, regions, and compliance information · `packages/models`
- [Database](https://github.com/theopenco/llmgateway/tree/main/packages/db): postgres schema · Organizations, projects, keys, billing, logs, metrics, discounts, and webhooks schema plus helpers · `packages/db`
- [Cache](https://github.com/theopenco/llmgateway/tree/main/packages/cache): redis cache · Redis-backed caching, SWR query support, and transient shared state · `packages/cache`
- [Shared utils](https://github.com/theopenco/llmgateway/tree/main/packages/shared): cross-app helpers · Routing config, token estimation, load balancing, URL safety, fees, and shared UI primitives · `packages/shared`
- [Instrumentation](https://github.com/theopenco/llmgateway/tree/main/packages/instrumentation): telemetry · Tracing, metrics, request lifecycle hooks, and sampling · `packages/instrumentation`

### Enterprise

Enterprise-only policy and admin extensions

- [Guardrails](https://github.com/theopenco/llmgateway/tree/main/ee/guardrails): policy engine · Enterprise policy, redaction, and content restriction enforcement · `ee/guardrails`
- [Audit](https://github.com/theopenco/llmgateway/tree/main/ee/audit): audit logging · Structured enterprise audit logging and schema · `ee/audit`
- [Admin](https://github.com/theopenco/llmgateway/tree/main/ee/admin): enterprise console · Internal operations console for orgs, providers, models, rate limits, discounts, and stats · `ee/admin`

### Operations

Deployment and runtime orchestration

- [Deploy](https://github.com/theopenco/llmgateway/tree/main/infra): runtime topology · Unified and split Docker, Compose, Helm, and startup scripts for the monorepo services · `infra`

## Connections

- Gateway → Chat flow: serves · Main inference path is implemented in the chat pipeline
- Gateway → Gateway APIs: exposes · Gateway also serves specialized modality and utility endpoints
- Chat flow → Actions: transforms via · Request normalization and provider-specific payload work is delegated to shared actions
- Chat flow → Models: resolves · Model selection and capability checks use shared model metadata
- Chat flow → Database: reads/writes · Logs, keys, routing config, and usage state persist in Postgres
- Chat flow → Cache: caches · Rate limiting, SWR data, and transient gateway state use Redis
- Chat flow → Instrumentation: emits · Tracing and metrics are recorded around request handling
- Gateway APIs → Actions: uses · Non-chat endpoints still share request transformation and provider logic
- Gateway APIs → Models: validates · Endpoint availability depends on model and provider capabilities
- Gateway → Guardrails: enforces · Enterprise policy checks can block or redact inference requests
- Gateway → Audit: logs · Security-relevant gateway events feed enterprise audit trails
- API → Database: persists · Control-plane data is managed through the shared database schema
- API → Cache: uses · Some API flows rely on shared Redis-backed caching
- API → Instrumentation: traces · API requests are instrumented for observability
- UI → API: consumes · Dashboard and account flows are backed by the control-plane API
- Playground → API: consumes · Playground identity, org, key, and plan flows use the API
- Code site → API: consumes · Coding-plan and profile flows depend on the backend API
- Docs → API: documents · Docs describe the public API and product behavior
- Worker → Database: aggregates · Background jobs read logs and usage data to build stats
- Worker → Cache: coordinates · Jobs and transient processing state can use Redis
- Worker → Instrumentation: reports · Worker execution is observable through shared telemetry
- Worker → API: supports · Async processing backs account and analytics experiences
- Admin → API: administers · Enterprise console surfaces and mutates platform state through backend APIs
- Admin → Database: queries · Admin dashboards read from shared operational data
- Admin → Audit: reviews · Audit records are part of the internal admin workflow
- Guardrails → Gateway: guards · Policy engine sits on the inference path for sensitive requests
- Guardrails → API: configures · Guardrail settings are managed from control-plane surfaces
- Deploy → Gateway: deploys · Deployment manifests include the gateway service
- Deploy → API: deploys · Deployment manifests include the API service
- Deploy → UI: deploys · Deployment manifests include the customer UI
- Deploy → Playground: deploys · Deployment manifests include the playground app
- Deploy → Code site: deploys · Deployment manifests include the coding site
- Deploy → Docs: deploys · Deployment manifests include the docs site
- Deploy → Worker: deploys · Deployment manifests include the background worker
- Deploy → Database: provisions · PostgreSQL is part of the bundled runtime topology
- Deploy → Cache: provisions · Redis is part of the bundled runtime topology