theopenco/llmgateway

1. This repository is a multi-app monorepo for an LLM gateway product, split into a hot-path inference layer and a control-plane/product layer. The main runtime pieces are a Hono-based backend API, a dedicated gateway service, several Next.js apps, a background worker, and shared packages for data, models, caching, and utilities.

2. The core data-plane is apps/gateway. It accepts OpenAI-compatible requests and also exposes specialized endpoints for embeddings, images, OCR, speech, video, and responses. Its job is to normalize requests, choose providers and fallback paths, translate provider-specific formats, and enforce routing, rate limits, and capability checks before sending traffic to external LLM providers.

3. The gateway’s outbound dependencies are the actual model providers: OpenAI, Anthropic, Google/Vertex, AWS Bedrock, Azure, DeepSeek, Mistral, xAI, etc. Provider/model metadata lives in packages/models, while request transformation and provider selection logic is concentrated in packages/actions and apps/gateway/src/chat/tools. That is the main abstraction boundary for “provider-agnostic” LLM access.

4. Persistence and operational state are centered on PostgreSQL and Redis. packages/db contains the schema, migrations, seed data, and query helpers for organizations, projects, API keys, logs, metrics, billing, discounts, and webhooks. Redis is used for caching, SWR-style query support, rate limiting, and transient gateway state. In diagrams, this is the shared state layer beneath the API, gateway, worker, and admin surfaces.

5. apps/api is the main backend control plane. It exposes endpoints for authentication, organizations, projects, keys, billing, analytics, audit logs, referrals, platform sessions, webhooks, and public configuration. This service is where the product’s account management and platform administration logic lives, distinct from the inference traffic handled by the gateway.

6. apps/ui, apps/playground, apps/code, and ee/admin are separate Next.js frontends with different audiences. apps/ui is the public marketing site plus the customer dashboard; apps/playground is the interactive product demo and authenticated sandbox; apps/code focuses on coding-agent/dev-plan content and user-facing plan management; ee/admin is an internal enterprise operations console. They all consume the backend API and shared data layer, but each has a distinct UX and domain scope.

7. apps/worker handles asynchronous and scheduled processing. It aggregates stats, syncs models, processes follow-up emails, and manages video-related jobs. This is the repository’s background-processing boundary: anything that should not block request/response flows is pushed here, usually reading from or writing to the same database and telemetry streams used by the API and gateway.

8. There is a clear separation between synchronous request handling and analytics/metadata processing. The gateway records logs and metrics about model usage, cost, latency, retries, and failures; the worker compacts or aggregates that data into project/global stats; the UI and admin dashboards query those aggregates for charts, tables, and cost/usage views. This is the main analytics data flow in the system.

9. Authentication and authorization are shared concerns but applied differently across surfaces. The API and dashboards use app-level auth, while the gateway also validates API tokens and provider credentials for inference requests. Enterprise and platform features such as org/team access, master keys, provider keys, billing, and guardrails sit behind these auth boundaries.

10. The ee/ directory adds enterprise-only modules and UI. The notable runtime pieces are ee/guardrails for policy and redaction enforcement, ee/audit for structured audit logging, and ee/admin for the internal admin dashboard. These are layered on top of the core OSS platform and share the same general data and identity model.

11. Docs and content are first-class but architecturally separate. apps/docs is a documentation site with MDX content and API/search/proxy routes, while apps/ui and apps/code also host large content libraries for blog posts, changelogs, guides, comparisons, and legal pages. These sites are mostly read-only content delivery, not part of the runtime LLM request path.

12. Deployment supports both split and unified topologies. The repo includes Docker Compose variants, unified and split Dockerfiles, Helm charts, and run scripts. In a diagram, the unified deployment bundles the gateway, API, frontends, worker, PostgreSQL, and Redis; the split deployment treats the gateway as a separate service from the control-plane apps.

13. Shared libraries are intentionally kept small and domain-focused. packages/shared contains cross-app utilities like routing config, token estimation, load balancing, URL safety, fees, video access, and shared UI primitives; packages/cache and packages/logger provide infrastructure helpers; packages/instrumentation provides tracing, metrics, and request lifecycle hooks. These packages are the glue that keeps the monorepo consistent across services.