How llm-d/llm-d-router works
llm-d Router is an LLM inference traffic router: proxies send requests to the Endpoint Picker (EPP), which uses endpoint state and scheduling/flow-control decisions to select placement. A separate coordinator supports multi-stage inference and a sidecar coordinates encode/prefill/decode transfers. The README documents InferenceObjective and InferenceModelRewrite APIs, proxy deployment modes, and Gateway API integration. Arrows are limited to README statements and source-index references or excerpts; unsampled plugin wiring is not inferred.
Main entry points: server.go Ext-proc handler · runner.go EPP runtime · pipeline.go Inference pipeline · main.go Coordinator runtime
Components 22
Endpoint Picker
- EPP runtime
cmd/epp/runner/runner.go - Ext-proc handler
pkg/epp/handlers/server.go - Endpoint data layer
pkg/epp/datalayer/manager.go - Request flow control
pkg/epp/flowcontrol/controller/controller.go - Scheduling plugins
pkg/epp/framework/plugins/scheduling - Request data plugins
pkg/epp/framework/plugins/requestcontrol/dataproducer
Request APIs
- Inference objectives
apix/v1alpha2/inferenceobjective_types.go - Model rewrites
apix/v1alpha2/inferencemodelrewrite_types.go - Kubernetes client tools
client-go/clientset/versioned/clientset.go
Coordinator
- Coordinator runtime
cmd/coordinator/main.go - HTTP server
pkg/coordinator/server/server.go - Inference pipeline
pkg/coordinator/pipeline/pipeline.go - Gateway client
pkg/coordinator/gateway/client.go - Async broker
pkg/coordinator/steps/asyncbroker/routes.go
Disaggregation
- Disaggregation sidecar
cmd/pd-sidecar/main.go - P/D and E/P/D transfer
pkg/sidecar/proxy/connector_shared_storage.go - KV event adapter
pkg/kvevents/engineadapter/vllm_adapter.go - Event subscribers
pkg/kvevents/subscriber_manager.go - KV prefix cache
pkg/kvcache/prefix_match.go
Observability
- EPP metrics
pkg/epp/metrics/llm_d_router_metrics.go - Coordinator metrics
pkg/coordinator/metrics/llm_d_coordinator_metrics.go
Other components
- Envoy / L7 proxy
Connections 24
- Envoy / L7 proxy → Ext-proc handler: sends ext-proc
README.md - Ext-proc handler → Endpoint data layer: reads endpoint state
pkg/epp/handlers/server.go - Ext-proc handler → Request flow control: submits requests
- EPP runtime → Endpoint data layer: initializes
cmd/epp/runner/runner.go - EPP runtime → Request flow control: initializes
cmd/epp/runner/runner.go - Ext-proc handler → EPP metrics: records request metrics
pkg/epp/handlers/server.go - Request flow control → EPP metrics: records flow metrics
pkg/epp/flowcontrol/controller/controller.go - Inference objectives → EPP runtime: configures scheduling
README.md - Model rewrites → EPP runtime: configures model rewriting
README.md - Kubernetes client tools → Inference objectives: accesses resources
client-go/clientset/versioned/clientset.go - Kubernetes client tools → Model rewrites: accesses resources
client-go/clientset/versioned/clientset.go - Coordinator runtime → HTTP server: starts server
cmd/coordinator/main.go - Coordinator runtime → Inference pipeline: builds pipeline
cmd/coordinator/main.go - Coordinator runtime → Gateway client: initializes client
cmd/coordinator/main.go - HTTP server → Inference pipeline: executes requests
pkg/coordinator/server/server.go - HTTP server → Gateway client: uses gateway client
pkg/coordinator/server/server.go - Inference pipeline → Coordinator metrics: records pipeline metrics
pkg/coordinator/pipeline/pipeline.go - Inference pipeline → Async broker: serves auxiliary routes
- Disaggregation sidecar → P/D and E/P/D transfer: starts proxy
cmd/pd-sidecar/main.go - P/D and E/P/D transfer → KV prefix cache: uses shared KV state
- KV event adapter → Event subscribers: feeds event pool
- Event subscribers → KV prefix cache: updates cache metrics
pkg/kvevents/subscriber_manager.go - Endpoint data layer → Request data plugins: loads plugins
- Ext-proc handler → Scheduling plugins: applies scheduling