How portkey-ai/gateway works
1. This repository is an AI gateway, not a single model service. Its main job is to accept OpenAI-style client requests, apply policy/routing/reliability logic, and forward them to one of many provider-specific adapters.
2. The runtime entrypoints are src/index.ts and src/start-server.ts, with deployment targets spanning Node.js, Docker, and Cloudflare Workers. wrangler.toml, Dockerfile, and docker-compose.yaml show that the same gateway can run in multiple environments.
3. The public API surface lives in src/handlers. These handlers cover the major LLM-related endpoints: chat/completions, completions, embeddings, models, files, fine-tuning, batches, images, speech/transcription/translation, messages, realtime, proxying, and retry behavior. That is the main ingress boundary for all traffic.
4. A request typically enters a handler, then passes through middleware layers in src/middlewares: request validation, caching, hooks, and logging. These are cross-cutting concerns applied before and/or after the provider call rather than part of any single provider implementation.
5. The gateway’s request orchestration lives in src/handlers/services and src/services. Key responsibilities there include request context setup, provider context selection, pre-request validation, response shaping, cache management, hook execution, and transformation of gateway configs into provider requests. conditionalRouter.ts is the main routing decision layer.
6. Provider integration is adapter-based. src/providers contains a large set of provider subdirectories, each wrapping a vendor’s API shape into the gateway’s normalized behavior. Many providers implement their own chatComplete, complete, embed, imageGenerate, batch, or audio methods, while shared base layers like open-ai-base and anthropic-base reduce duplication.
7. This adapter layer is the key boundary between “gateway logic” and “external model providers.” The gateway decides where to send traffic; the provider adapter handles auth, payload translation, endpoint semantics, and response normalization for that vendor.
8. Routing features like retries, fallbacks, load balancing, and conditional routing are first-class concerns, not add-ons. They are implemented around the provider call path, so a single user request can fan out across multiple providers/models depending on policy, health, or config.
9. Caching is also a core subsystem. src/shared/services/cache supports multiple backends: memory, file, Redis, and Cloudflare KV. That means cache behavior is abstracted from the request flow and can be swapped by deployment environment.
10. Guardrails and validation are extensibility points. The plugins directory contains many rule packages for content moderation, PII detection, prompt protection, output checks, and vendor-specific safety integrations. These plugins are referenced by config and are evaluated as part of the gateway’s request/response policy pipeline.
11. Observability is built into the core path via logging services and an APM module. The README and the src/handlers/services/logsService.ts / src/apm areas indicate the gateway is designed to capture request metadata, latency, errors, and routing outcomes for operational visibility.
12. Realtime and multimodal support are separate but integrated concerns. There are dedicated handlers for realtime and WebSocket utilities, plus provider adapters for speech, transcription, translation, image generation, and embeddings. This means the gateway is not limited to text chat.
13. Configuration and environment shape behavior heavily. conf.json / conf.example.json, initializeSettings.ts, and the deployment files define gateway behavior, provider keys, routing rules, cache settings, and feature toggles. The gateway is therefore config-driven rather than hardcoded around one backend.
14. The src/public static area and the large cookbook directory are supporting layers, not core runtime logic. They provide the gateway console and usage examples/integrations, while the actual production architecture is concentrated in handlers, middleware, services, provider adapters, and plugins.
Main entry points: services Handler Services · providers/index.ts Provider Registry · src/index.ts Index · handlers Handlers
Components 23
Runtime
Entrypoints and deployment targets
- Index · Primary runtime entrypoint
src/index.ts - Start Server · Server bootstrap entrypoint
src/start-server.ts - Docker · Container runtime definition
Dockerfile - Compose · Local multi-service deployment
docker-compose.yaml - Workers · Cloudflare Workers deployment config
wrangler.toml
Gateway Core
Ingress, orchestration, and cross-cutting request flow
- Handlers · Public request endpoints for LLM and media APIs
src/handlers - Handler Services · Request context, provider context, logging, hooks, cache, validation
src/handlers/services - Middleware · Validation, cache, hooks, and logging layers
src/middlewares - Router · Conditional routing, retries, fallbacks, and load balancing decisions
src/services/conditionalRouter.ts - Transform · Gateway config to provider request translation
src/services/transformToProviderRequest.ts - Realtime Parser · Realtime event parsing and stream handling
src/services/realtimeLlmEventParser.ts
Providers
Adapter layer for external model vendors
- Provider Registry · Provider module aggregation and export boundary
src/providers/index.ts - Base Adapters · Shared OpenAI and Anthropic compatibility layers
src/providers/open-ai-base - OpenAI · Representative OpenAI-compatible provider adapter
src/providers/openai - Anthropic · Representative non-OpenAI provider adapter
src/providers/anthropic - Bedrock · Provider adapter with batch, files, and multimodal methods
src/providers/bedrock - Vertex AI · Provider adapter with generation config and batch support
src/providers/google-vertex-ai
Policy & Extensions
Caching, guardrails, validation, and plugins
Connections 34
- Index → Handlers: serves · Routes incoming traffic into API handlers
- Start Server → Handlers: boots · Starts server runtime around the handler surface
- Index → Provider Registry: loads · Initializes provider registry during startup
- Index → Cache: configures · Selects cache backend from deployment settings
- Index → Logger: initializes · Sets up shared logging for request flow
- Index → APM: initializes · Enables telemetry collection in runtime
- Docker → Start Server: runs · Container entrypoint executes server bootstrap
- Compose → Docker: orchestrates · Local deployment composes containerized gateway
- Workers → Index: deploys · Workers runtime targets the gateway entrypoint
- Handlers → Handler Services: delegates · Handlers offload orchestration to service layer
- Handlers → Middleware: passes through · Request flow crosses middleware layers
- Handler Services → Router: routes · Routing decisions determine provider selection and fallbacks
- Handler Services → Transform: shapes · Transforms gateway request into provider-specific payloads
- Handler Services → Realtime Parser: streams · Realtime and websocket flows use event parsing
- Handler Services → Cache: uses · Applies request and response caching
- Handler Services → Plugins: evaluates · Runs guardrails and policy plugins around requests
- Handler Services → Logger: logs · Captures request and routing metadata
- Handler Services → APM: traces · Sends latency and error telemetry
- Middleware → Cache: wraps · Cache middleware fronts the shared cache system
- Middleware → Plugins: hooks · Hooks and validation middleware enforce policy
- Provider Registry → Base Adapters: shares · Registry includes compatibility base layers
- Provider Registry → OpenAI: exports · Registry exposes provider adapters
- Provider Registry → Anthropic: exports · Registry exposes provider adapters
- Provider Registry → Bedrock: exports · Registry exposes provider adapters
- Provider Registry → Vertex AI: exports · Registry exposes provider adapters
- Router → Provider Registry: selects · Chooses provider adapter based on config and policy
- Transform → Base Adapters: adapts · Normalizes request shape toward provider contracts
- OpenAI → Base Adapters: extends · Uses OpenAI-compatible base helpers
- Anthropic → Base Adapters: extends · Uses Anthropic compatibility base helpers
- Bedrock → Base Adapters: extends · Builds on shared adapter conventions
- Vertex AI → Base Adapters: extends · Builds on shared adapter conventions
- Plugins → Logger: reports · Policy checks emit structured logs
- Public UI → Handlers: consumes · Console talks to gateway API endpoints
- Cookbook → Handlers: documents · Examples describe API usage patterns