# llm-d/llm-d-router architecture

> Architecture diagram of the GitHub repository llm-d/llm-d-router, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/llm-d/llm-d-router
- Repository: https://github.com/llm-d/llm-d-router

## Overview

llm-d Router is an LLM inference traffic router: proxies send requests to the Endpoint Picker (EPP), which uses endpoint state and scheduling/flow-control decisions to select placement. A separate coordinator supports multi-stage inference and a sidecar coordinates encode/prefill/decode transfers. The README documents InferenceObjective and InferenceModelRewrite APIs, proxy deployment modes, and Gateway API integration. Arrows are limited to README statements and source-index references or excerpts; unsampled plugin wiring is not inferred.

Main entry points: [`pkg/epp/handlers/server.go`](https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/handlers/server.go) (Ext-proc handler), [`cmd/epp/runner/runner.go`](https://github.com/llm-d/llm-d-router/blob/main/cmd/epp/runner/runner.go) (EPP runtime), [`pkg/coordinator/pipeline/pipeline.go`](https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/pipeline/pipeline.go) (Inference pipeline), [`cmd/coordinator/main.go`](https://github.com/llm-d/llm-d-router/blob/main/cmd/coordinator/main.go) (Coordinator runtime)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_epp["Endpoint Picker"]
  node_epp_entry["EPP runtime<br/>[runner.go]"]
  node_extproc["Ext-proc handler<br/>[server.go]"]
  node_data["Endpoint data layer<br/>[manager.go]"]
  node_flow["Request flow control<br/>[controller.go]"]
  node_plugins["Scheduling plugins"]
  node_dataproducers["Request data plugins"]
end

subgraph group_api["Request APIs"]
  node_objectives["Inference objectives"]
  node_rewrites["Model rewrites"]
  node_client["Kubernetes client tools<br/>[clientset.go]"]
end

subgraph group_coord["Coordinator"]
  node_coordinator["Coordinator runtime<br/>[main.go]"]
  node_coord_server["HTTP server<br/>[server.go]"]
  node_pipeline["Inference pipeline<br/>[pipeline.go]"]
  node_gateway["Gateway client<br/>[client.go]"]
  node_async["Async broker<br/>[routes.go]"]
end

subgraph group_disagg["Disaggregation"]
  node_sidecar["Disaggregation sidecar<br/>[main.go]"]
  node_transfer["P/D and E/P/D transfer"]
  node_kv_events["KV event adapter<br/>[vllm_adapter.go]"]
  node_subscribers["Event subscribers"]
  node_cache[("KV prefix cache<br/>[prefix_match.go]")]
end

subgraph group_telemetry["Observability"]
  node_epp_metrics["EPP metrics"]
  node_metrics["Coordinator metrics"]
end

node_proxy(("Envoy / L7 proxy"))

node_proxy -->|"sends ext-proc"| node_extproc
node_extproc -->|"reads endpoint state"| node_data
node_extproc -.->|"submits requests"| node_flow
node_epp_entry -->|"initializes"| node_data
node_epp_entry -->|"initializes"| node_flow
node_extproc -->|"records request metrics"| node_epp_metrics
node_flow -->|"records flow metrics"| node_epp_metrics
node_objectives -->|"configures scheduling"| node_epp_entry
node_rewrites -->|"configures model rewriting"| node_epp_entry
node_client -->|"accesses resources"| node_objectives
node_client -->|"accesses resources"| node_rewrites
node_coordinator -->|"starts server"| node_coord_server
node_coordinator -->|"builds pipeline"| node_pipeline
node_coordinator -->|"initializes client"| node_gateway
node_coord_server -->|"executes requests"| node_pipeline
node_coord_server -->|"uses gateway client"| node_gateway
node_pipeline -->|"records pipeline metrics"| node_metrics
node_pipeline -.->|"serves auxiliary routes"| node_async
node_sidecar -->|"starts proxy"| node_transfer
node_transfer -.->|"uses shared KV state"| node_cache
node_kv_events -.->|"feeds event pool"| node_subscribers
node_subscribers -->|"updates cache metrics"| node_cache
node_data -.->|"loads plugins"| node_dataproducers
node_extproc -.->|"applies scheduling"| node_plugins

click node_epp_entry "https://github.com/llm-d/llm-d-router/blob/main/cmd/epp/runner/runner.go"
click node_extproc "https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/handlers/server.go"
click node_data "https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/datalayer/manager.go"
click node_flow "https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/flowcontrol/controller/controller.go"
click node_plugins "https://github.com/llm-d/llm-d-router/tree/main/pkg/epp/framework/plugins/scheduling"
click node_dataproducers "https://github.com/llm-d/llm-d-router/tree/main/pkg/epp/framework/plugins/requestcontrol/dataproducer"
click node_epp_metrics "https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/metrics/llm_d_router_metrics.go"
click node_objectives "https://github.com/llm-d/llm-d-router/blob/main/apix/v1alpha2/inferenceobjective_types.go"
click node_rewrites "https://github.com/llm-d/llm-d-router/blob/main/apix/v1alpha2/inferencemodelrewrite_types.go"
click node_client "https://github.com/llm-d/llm-d-router/blob/main/client-go/clientset/versioned/clientset.go"
click node_coordinator "https://github.com/llm-d/llm-d-router/blob/main/cmd/coordinator/main.go"
click node_coord_server "https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/server/server.go"
click node_pipeline "https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/pipeline/pipeline.go"
click node_gateway "https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/gateway/client.go"
click node_async "https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/steps/asyncbroker/routes.go"
click node_sidecar "https://github.com/llm-d/llm-d-router/blob/main/cmd/pd-sidecar/main.go"
click node_transfer "https://github.com/llm-d/llm-d-router/blob/main/pkg/sidecar/proxy/connector_shared_storage.go"
click node_kv_events "https://github.com/llm-d/llm-d-router/blob/main/pkg/kvevents/engineadapter/vllm_adapter.go"
click node_subscribers "https://github.com/llm-d/llm-d-router/blob/main/pkg/kvevents/subscriber_manager.go"
click node_cache "https://github.com/llm-d/llm-d-router/blob/main/pkg/kvcache/prefix_match.go"
click node_metrics "https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/metrics/llm_d_coordinator_metrics.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_epp_entry,node_extproc,node_data,node_flow,node_plugins,node_dataproducers toneBlue
class node_objectives,node_rewrites,node_client toneAmber
class node_coordinator,node_coord_server,node_pipeline,node_gateway,node_async toneMint
class node_sidecar,node_transfer,node_kv_events,node_subscribers,node_cache toneRose
class node_epp_metrics,node_metrics,node_proxy toneIndigo
```

## Components

### Endpoint Picker

- [EPP runtime](https://github.com/llm-d/llm-d-router/blob/main/cmd/epp/runner/runner.go): `cmd/epp/runner/runner.go`
- [Ext-proc handler](https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/handlers/server.go): `pkg/epp/handlers/server.go`
- [Endpoint data layer](https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/datalayer/manager.go): `pkg/epp/datalayer/manager.go`
- [Request flow control](https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/flowcontrol/controller/controller.go): `pkg/epp/flowcontrol/controller/controller.go`
- [Scheduling plugins](https://github.com/llm-d/llm-d-router/tree/main/pkg/epp/framework/plugins/scheduling): `pkg/epp/framework/plugins/scheduling`
- [Request data plugins](https://github.com/llm-d/llm-d-router/tree/main/pkg/epp/framework/plugins/requestcontrol/dataproducer): `pkg/epp/framework/plugins/requestcontrol/dataproducer`

### Request APIs

- [Inference objectives](https://github.com/llm-d/llm-d-router/blob/main/apix/v1alpha2/inferenceobjective_types.go): `apix/v1alpha2/inferenceobjective_types.go`
- [Model rewrites](https://github.com/llm-d/llm-d-router/blob/main/apix/v1alpha2/inferencemodelrewrite_types.go): `apix/v1alpha2/inferencemodelrewrite_types.go`
- [Kubernetes client tools](https://github.com/llm-d/llm-d-router/blob/main/client-go/clientset/versioned/clientset.go): `client-go/clientset/versioned/clientset.go`

### Coordinator

- [Coordinator runtime](https://github.com/llm-d/llm-d-router/blob/main/cmd/coordinator/main.go): `cmd/coordinator/main.go`
- [HTTP server](https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/server/server.go): `pkg/coordinator/server/server.go`
- [Inference pipeline](https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/pipeline/pipeline.go): `pkg/coordinator/pipeline/pipeline.go`
- [Gateway client](https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/gateway/client.go): `pkg/coordinator/gateway/client.go`
- [Async broker](https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/steps/asyncbroker/routes.go): `pkg/coordinator/steps/asyncbroker/routes.go`

### Disaggregation

- [Disaggregation sidecar](https://github.com/llm-d/llm-d-router/blob/main/cmd/pd-sidecar/main.go): `cmd/pd-sidecar/main.go`
- [P/D and E/P/D transfer](https://github.com/llm-d/llm-d-router/blob/main/pkg/sidecar/proxy/connector_shared_storage.go): `pkg/sidecar/proxy/connector_shared_storage.go`
- [KV event adapter](https://github.com/llm-d/llm-d-router/blob/main/pkg/kvevents/engineadapter/vllm_adapter.go): `pkg/kvevents/engineadapter/vllm_adapter.go`
- [Event subscribers](https://github.com/llm-d/llm-d-router/blob/main/pkg/kvevents/subscriber_manager.go): `pkg/kvevents/subscriber_manager.go`
- [KV prefix cache](https://github.com/llm-d/llm-d-router/blob/main/pkg/kvcache/prefix_match.go): `pkg/kvcache/prefix_match.go`

### Observability

- [EPP metrics](https://github.com/llm-d/llm-d-router/blob/main/pkg/epp/metrics/llm_d_router_metrics.go): `pkg/epp/metrics/llm_d_router_metrics.go`
- [Coordinator metrics](https://github.com/llm-d/llm-d-router/blob/main/pkg/coordinator/metrics/llm_d_coordinator_metrics.go): `pkg/coordinator/metrics/llm_d_coordinator_metrics.go`

### Other components

- **Envoy / L7 proxy**

## Connections

- Envoy / L7 proxy → Ext-proc handler: sends ext-proc (evidence: `README.md`)
- Ext-proc handler → Endpoint data layer: reads endpoint state (evidence: `pkg/epp/handlers/server.go`)
- Ext-proc handler → Request flow control: submits requests
- EPP runtime → Endpoint data layer: initializes (evidence: `cmd/epp/runner/runner.go`)
- EPP runtime → Request flow control: initializes (evidence: `cmd/epp/runner/runner.go`)
- Ext-proc handler → EPP metrics: records request metrics (evidence: `pkg/epp/handlers/server.go`)
- Request flow control → EPP metrics: records flow metrics (evidence: `pkg/epp/flowcontrol/controller/controller.go`)
- Inference objectives → EPP runtime: configures scheduling (evidence: `README.md`)
- Model rewrites → EPP runtime: configures model rewriting (evidence: `README.md`)
- Kubernetes client tools → Inference objectives: accesses resources (evidence: `client-go/clientset/versioned/clientset.go`)
- Kubernetes client tools → Model rewrites: accesses resources (evidence: `client-go/clientset/versioned/clientset.go`)
- Coordinator runtime → HTTP server: starts server (evidence: `cmd/coordinator/main.go`)
- Coordinator runtime → Inference pipeline: builds pipeline (evidence: `cmd/coordinator/main.go`)
- Coordinator runtime → Gateway client: initializes client (evidence: `cmd/coordinator/main.go`)
- HTTP server → Inference pipeline: executes requests (evidence: `pkg/coordinator/server/server.go`)
- HTTP server → Gateway client: uses gateway client (evidence: `pkg/coordinator/server/server.go`)
- Inference pipeline → Coordinator metrics: records pipeline metrics (evidence: `pkg/coordinator/pipeline/pipeline.go`)
- Inference pipeline → Async broker: serves auxiliary routes
- Disaggregation sidecar → P/D and E/P/D transfer: starts proxy (evidence: `cmd/pd-sidecar/main.go`)
- P/D and E/P/D transfer → KV prefix cache: uses shared KV state
- KV event adapter → Event subscribers: feeds event pool
- Event subscribers → KV prefix cache: updates cache metrics (evidence: `pkg/kvevents/subscriber_manager.go`)
- Endpoint data layer → Request data plugins: loads plugins
- Ext-proc handler → Scheduling plugins: applies scheduling
