# portkey-ai/gateway architecture

> Architecture diagram of the GitHub repository portkey-ai/gateway, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/portkey-ai/gateway
- Repository: https://github.com/portkey-ai/gateway

## Overview

1. This repository is an AI gateway, not a single model service. Its main job is to accept OpenAI-style client requests, apply policy/routing/reliability logic, and forward them to one of many provider-specific adapters.

2. The runtime entrypoints are `src/index.ts` and `src/start-server.ts`, with deployment targets spanning Node.js, Docker, and Cloudflare Workers. `wrangler.toml`, `Dockerfile`, and `docker-compose.yaml` show that the same gateway can run in multiple environments.

3. The public API surface lives in `src/handlers`. These handlers cover the major LLM-related endpoints: chat/completions, completions, embeddings, models, files, fine-tuning, batches, images, speech/transcription/translation, messages, realtime, proxying, and retry behavior. That is the main ingress boundary for all traffic.

4. A request typically enters a handler, then passes through middleware layers in `src/middlewares`: request validation, caching, hooks, and logging. These are cross-cutting concerns applied before and/or after the provider call rather than part of any single provider implementation.

5. The gateway’s request orchestration lives in `src/handlers/services` and `src/services`. Key responsibilities there include request context setup, provider context selection, pre-request validation, response shaping, cache management, hook execution, and transformation of gateway configs into provider requests. `conditionalRouter.ts` is the main routing decision layer.

6. Provider integration is adapter-based. `src/providers` contains a large set of provider subdirectories, each wrapping a vendor’s API shape into the gateway’s normalized behavior. Many providers implement their own `chatComplete`, `complete`, `embed`, `imageGenerate`, batch, or audio methods, while shared base layers like `open-ai-base` and `anthropic-base` reduce duplication.

7. This adapter layer is the key boundary between “gateway logic” and “external model providers.” The gateway decides where to send traffic; the provider adapter handles auth, payload translation, endpoint semantics, and response normalization for that vendor.

8. Routing features like retries, fallbacks, load balancing, and conditional routing are first-class concerns, not add-ons. They are implemented around the provider call path, so a single user request can fan out across multiple providers/models depending on policy, health, or config.

9. Caching is also a core subsystem. `src/shared/services/cache` supports multiple backends: memory, file, Redis, and Cloudflare KV. That means cache behavior is abstracted from the request flow and can be swapped by deployment environment.

10. Guardrails and validation are extensibility points. The `plugins` directory contains many rule packages for content moderation, PII detection, prompt protection, output checks, and vendor-specific safety integrations. These plugins are referenced by config and are evaluated as part of the gateway’s request/response policy pipeline.

11. Observability is built into the core path via logging services and an APM module. The README and the `src/handlers/services/logsService.ts` / `src/apm` areas indicate the gateway is designed to capture request metadata, latency, errors, and routing outcomes for operational visibility.

12. Realtime and multimodal support are separate but integrated concerns. There are dedicated handlers for realtime and WebSocket utilities, plus provider adapters for speech, transcription, translation, image generation, and embeddings. This means the gateway is not limited to text chat.

13. Configuration and environment shape behavior heavily. `conf.json` / `conf.example.json`, `initializeSettings.ts`, and the deployment files define gateway behavior, provider keys, routing rules, cache settings, and feature toggles. The gateway is therefore config-driven rather than hardcoded around one backend.

14. The `src/public` static area and the large `cookbook` directory are supporting layers, not core runtime logic. They provide the gateway console and usage examples/integrations, while the actual production architecture is concentrated in handlers, middleware, services, provider adapters, and plugins.

Main entry points: [`src/handlers/services`](https://github.com/portkey-ai/gateway/tree/main/src/handlers/services) (Handler Services), [`src/providers/index.ts`](https://github.com/portkey-ai/gateway/blob/main/src/providers/index.ts) (Provider Registry), [`src/index.ts`](https://github.com/portkey-ai/gateway/blob/main/src/index.ts) (Index), [`src/handlers`](https://github.com/portkey-ai/gateway/tree/main/src/handlers) (Handlers)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_src_index["Index<br/>entrypoint<br/>[index.ts]"]
  node_src_start_server["Start Server<br/>entrypoint<br/>[start-server.ts]"]
  node_dockerfile["Docker<br/>deploy"]
  node_docker_compose["Compose<br/>deploy"]
  node_wrangler["Workers<br/>deploy<br/>[wrangler.toml]"]
end

subgraph group_gateway["Gateway Core"]
  node_handlers["Handlers<br/>api-surface"]
  node_handler_services["Handler Services<br/>orchestration"]
  node_middlewares["Middleware<br/>cross-cutting"]
  node_conditional_router["Router<br/>routing"]
  node_request_transform["Transform<br/>request-shaping"]
  node_realtime_parser["Realtime Parser<br/>streaming"]
end

subgraph group_providers["Providers"]
  node_provider_index["Provider Registry<br/>adapter-registry<br/>[index.ts]"]
  node_provider_base["Base Adapters<br/>adapter-base"]
  node_provider_openai["OpenAI<br/>adapter"]
  node_provider_anthropic["Anthropic<br/>adapter"]
  node_provider_bedrock["Bedrock<br/>adapter"]
  node_provider_vertex["Vertex AI<br/>adapter"]
end

subgraph group_policy["Policy & Extensions"]
  node_cache[("Cache<br/>state")]
  node_plugins["Plugins<br/>guardrails"]
  node_apm["APM<br/>observability<br/>[index.ts]"]
  node_logger["Logger<br/>observability<br/>[logger.ts]"]
end

subgraph group_support["Support"]
  node_public_console["Public UI<br/>static-ui<br/>[index.html]"]
  node_cookbook["Cookbook<br/>examples"]
end

node_src_index -->|"serves"| node_handlers
node_src_start_server -->|"boots"| node_handlers
node_src_index -.->|"loads"| node_provider_index
node_src_index -.->|"configures"| node_cache
node_src_index -.->|"initializes"| node_logger
node_src_index -.->|"initializes"| node_apm
node_dockerfile -->|"runs"| node_src_start_server
node_docker_compose -.->|"orchestrates"| node_dockerfile
node_wrangler -.->|"deploys"| node_src_index
node_handlers -->|"delegates"| node_handler_services
node_handlers -->|"passes through"| node_middlewares
node_handler_services -->|"routes"| node_conditional_router
node_handler_services -->|"shapes"| node_request_transform
node_handler_services -->|"streams"| node_realtime_parser
node_handler_services -->|"uses"| node_cache
node_handler_services -->|"evaluates"| node_plugins
node_handler_services -->|"logs"| node_logger
node_handler_services -->|"traces"| node_apm
node_middlewares -->|"wraps"| node_cache
node_middlewares -->|"hooks"| node_plugins
node_provider_index -.->|"shares"| node_provider_base
node_provider_index -->|"exports"| node_provider_openai
node_provider_index -->|"exports"| node_provider_anthropic
node_provider_index -->|"exports"| node_provider_bedrock
node_provider_index -->|"exports"| node_provider_vertex
node_conditional_router -->|"selects"| node_provider_index
node_request_transform -->|"adapts"| node_provider_base
node_provider_openai -.->|"extends"| node_provider_base
node_provider_anthropic -.->|"extends"| node_provider_base
node_provider_bedrock -.->|"extends"| node_provider_base
node_provider_vertex -.->|"extends"| node_provider_base
node_plugins -.->|"reports"| node_logger
node_public_console -.->|"consumes"| node_handlers
node_cookbook -.->|"documents"| node_handlers

click node_src_index "https://github.com/portkey-ai/gateway/blob/main/src/index.ts"
click node_src_start_server "https://github.com/portkey-ai/gateway/blob/main/src/start-server.ts"
click node_dockerfile "https://github.com/portkey-ai/gateway/tree/main/Dockerfile"
click node_docker_compose "https://github.com/portkey-ai/gateway/blob/main/docker-compose.yaml"
click node_wrangler "https://github.com/portkey-ai/gateway/blob/main/wrangler.toml"
click node_handlers "https://github.com/portkey-ai/gateway/tree/main/src/handlers"
click node_handler_services "https://github.com/portkey-ai/gateway/tree/main/src/handlers/services"
click node_middlewares "https://github.com/portkey-ai/gateway/tree/main/src/middlewares"
click node_conditional_router "https://github.com/portkey-ai/gateway/blob/main/src/services/conditionalRouter.ts"
click node_request_transform "https://github.com/portkey-ai/gateway/blob/main/src/services/transformToProviderRequest.ts"
click node_realtime_parser "https://github.com/portkey-ai/gateway/blob/main/src/services/realtimeLlmEventParser.ts"
click node_cache "https://github.com/portkey-ai/gateway/tree/main/src/shared/services/cache"
click node_plugins "https://github.com/portkey-ai/gateway/tree/main/plugins"
click node_apm "https://github.com/portkey-ai/gateway/blob/main/src/apm/index.ts"
click node_logger "https://github.com/portkey-ai/gateway/blob/main/src/shared/utils/logger.ts"
click node_provider_index "https://github.com/portkey-ai/gateway/blob/main/src/providers/index.ts"
click node_provider_base "https://github.com/portkey-ai/gateway/tree/main/src/providers/open-ai-base"
click node_provider_openai "https://github.com/portkey-ai/gateway/tree/main/src/providers/openai"
click node_provider_anthropic "https://github.com/portkey-ai/gateway/tree/main/src/providers/anthropic"
click node_provider_bedrock "https://github.com/portkey-ai/gateway/tree/main/src/providers/bedrock"
click node_provider_vertex "https://github.com/portkey-ai/gateway/tree/main/src/providers/google-vertex-ai"
click node_public_console "https://github.com/portkey-ai/gateway/blob/main/src/public/index.html"
click node_cookbook "https://github.com/portkey-ai/gateway/tree/main/cookbook"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_src_index,node_src_start_server,node_dockerfile,node_docker_compose,node_wrangler toneBlue
class node_handlers,node_handler_services,node_middlewares,node_conditional_router,node_request_transform,node_realtime_parser toneAmber
class node_provider_index,node_provider_base,node_provider_openai,node_provider_anthropic,node_provider_bedrock,node_provider_vertex toneMint
class node_cache,node_plugins,node_apm,node_logger toneRose
class node_public_console,node_cookbook toneIndigo
```

## Components

### Runtime

Entrypoints and deployment targets

- [Index](https://github.com/portkey-ai/gateway/blob/main/src/index.ts): entrypoint · Primary runtime entrypoint · `src/index.ts`
- [Start Server](https://github.com/portkey-ai/gateway/blob/main/src/start-server.ts): entrypoint · Server bootstrap entrypoint · `src/start-server.ts`
- [Docker](https://github.com/portkey-ai/gateway/tree/main/Dockerfile): deploy · Container runtime definition · `Dockerfile`
- [Compose](https://github.com/portkey-ai/gateway/blob/main/docker-compose.yaml): deploy · Local multi-service deployment · `docker-compose.yaml`
- [Workers](https://github.com/portkey-ai/gateway/blob/main/wrangler.toml): deploy · Cloudflare Workers deployment config · `wrangler.toml`

### Gateway Core

Ingress, orchestration, and cross-cutting request flow

- [Handlers](https://github.com/portkey-ai/gateway/tree/main/src/handlers): api-surface · Public request endpoints for LLM and media APIs · `src/handlers`
- [Handler Services](https://github.com/portkey-ai/gateway/tree/main/src/handlers/services): orchestration · Request context, provider context, logging, hooks, cache, validation · `src/handlers/services`
- [Middleware](https://github.com/portkey-ai/gateway/tree/main/src/middlewares): cross-cutting · Validation, cache, hooks, and logging layers · `src/middlewares`
- [Router](https://github.com/portkey-ai/gateway/blob/main/src/services/conditionalRouter.ts): routing · Conditional routing, retries, fallbacks, and load balancing decisions · `src/services/conditionalRouter.ts`
- [Transform](https://github.com/portkey-ai/gateway/blob/main/src/services/transformToProviderRequest.ts): request-shaping · Gateway config to provider request translation · `src/services/transformToProviderRequest.ts`
- [Realtime Parser](https://github.com/portkey-ai/gateway/blob/main/src/services/realtimeLlmEventParser.ts): streaming · Realtime event parsing and stream handling · `src/services/realtimeLlmEventParser.ts`

### Providers

Adapter layer for external model vendors

- [Provider Registry](https://github.com/portkey-ai/gateway/blob/main/src/providers/index.ts): adapter-registry · Provider module aggregation and export boundary · `src/providers/index.ts`
- [Base Adapters](https://github.com/portkey-ai/gateway/tree/main/src/providers/open-ai-base): adapter-base · Shared OpenAI and Anthropic compatibility layers · `src/providers/open-ai-base`
- [OpenAI](https://github.com/portkey-ai/gateway/tree/main/src/providers/openai): adapter · Representative OpenAI-compatible provider adapter · `src/providers/openai`
- [Anthropic](https://github.com/portkey-ai/gateway/tree/main/src/providers/anthropic): adapter · Representative non-OpenAI provider adapter · `src/providers/anthropic`
- [Bedrock](https://github.com/portkey-ai/gateway/tree/main/src/providers/bedrock): adapter · Provider adapter with batch, files, and multimodal methods · `src/providers/bedrock`
- [Vertex AI](https://github.com/portkey-ai/gateway/tree/main/src/providers/google-vertex-ai): adapter · Provider adapter with generation config and batch support · `src/providers/google-vertex-ai`

### Policy & Extensions

Caching, guardrails, validation, and plugins

- [Cache](https://github.com/portkey-ai/gateway/tree/main/src/shared/services/cache): state · Shared cache abstraction across deployments · `src/shared/services/cache`
- [Plugins](https://github.com/portkey-ai/gateway/tree/main/plugins): guardrails · Content, PII, prompt, and response policy plugins · `plugins`
- [APM](https://github.com/portkey-ai/gateway/blob/main/src/apm/index.ts): observability · Application performance and request telemetry · `src/apm/index.ts`
- [Logger](https://github.com/portkey-ai/gateway/blob/main/src/shared/utils/logger.ts): observability · Shared logging utility · `src/shared/utils/logger.ts`

### Support

Static assets, docs, and examples

- [Public UI](https://github.com/portkey-ai/gateway/blob/main/src/public/index.html): static-ui · Static gateway console and front-end entrypoint · `src/public/index.html`
- [Cookbook](https://github.com/portkey-ai/gateway/tree/main/cookbook): examples · Usage guides and integrations · `cookbook`

## Connections

- Index → Handlers: serves · Routes incoming traffic into API handlers
- Start Server → Handlers: boots · Starts server runtime around the handler surface
- Index → Provider Registry: loads · Initializes provider registry during startup
- Index → Cache: configures · Selects cache backend from deployment settings
- Index → Logger: initializes · Sets up shared logging for request flow
- Index → APM: initializes · Enables telemetry collection in runtime
- Docker → Start Server: runs · Container entrypoint executes server bootstrap
- Compose → Docker: orchestrates · Local deployment composes containerized gateway
- Workers → Index: deploys · Workers runtime targets the gateway entrypoint
- Handlers → Handler Services: delegates · Handlers offload orchestration to service layer
- Handlers → Middleware: passes through · Request flow crosses middleware layers
- Handler Services → Router: routes · Routing decisions determine provider selection and fallbacks
- Handler Services → Transform: shapes · Transforms gateway request into provider-specific payloads
- Handler Services → Realtime Parser: streams · Realtime and websocket flows use event parsing
- Handler Services → Cache: uses · Applies request and response caching
- Handler Services → Plugins: evaluates · Runs guardrails and policy plugins around requests
- Handler Services → Logger: logs · Captures request and routing metadata
- Handler Services → APM: traces · Sends latency and error telemetry
- Middleware → Cache: wraps · Cache middleware fronts the shared cache system
- Middleware → Plugins: hooks · Hooks and validation middleware enforce policy
- Provider Registry → Base Adapters: shares · Registry includes compatibility base layers
- Provider Registry → OpenAI: exports · Registry exposes provider adapters
- Provider Registry → Anthropic: exports · Registry exposes provider adapters
- Provider Registry → Bedrock: exports · Registry exposes provider adapters
- Provider Registry → Vertex AI: exports · Registry exposes provider adapters
- Router → Provider Registry: selects · Chooses provider adapter based on config and policy
- Transform → Base Adapters: adapts · Normalizes request shape toward provider contracts
- OpenAI → Base Adapters: extends · Uses OpenAI-compatible base helpers
- Anthropic → Base Adapters: extends · Uses Anthropic compatibility base helpers
- Bedrock → Base Adapters: extends · Builds on shared adapter conventions
- Vertex AI → Base Adapters: extends · Builds on shared adapter conventions
- Plugins → Logger: reports · Policy checks emit structured logs
- Public UI → Handlers: consumes · Console talks to gateway API endpoints
- Cookbook → Handlers: documents · Examples describe API usage patterns
