How kubernetes-sigs/gateway-api-inference-extension works
This repository maintains the InferencePool Kubernetes API, a lightweight Envoy external-processing endpoint picker (LWEPP), generated Go client support, and conformance tests. The main runtime path is an inference request arriving through Envoy’s ext-proc integration, being handled by LWEPP and selecting a backend endpoint; separately, LWEPP watches pool and pod state through Kubernetes controllers and a datastore. The README documents the broader inference-gateway role and conformance scope. Some controller, datastore, and request-handling internals were not sampled, so those links are omitted where the available evidence does not establish them.
Main entry points: inferencepool_reconciler.go Pool reconciler · clientset.go Versioned clientset · datastore.go Endpoint datastore · inferencepool_types.go InferencePool API
Components 20
LWEPP runtime
- LWEPP process
cmd/lwepp/main.go - Controller manager
pkg/lwepp/server/controller_manager.go - Streaming handler
pkg/lwepp/handlers/server.go - Endpoint picker
pkg/lwepp/handlers/server.go - Endpoint datastore
pkg/lwepp/datastore/datastore.go - Pool reconciler
pkg/lwepp/controller/inferencepool_reconciler.go - Pod reconciler
pkg/lwepp/controller/pod_reconciler.go
Kubernetes APIs
- InferencePool API
api/v1/inferencepool_types.go - Pool import API
apix/v1alpha1/inferencepoolimport_types.go
Go client support
- Versioned clientset
client-go/clientset/versioned/clientset.go - Pool informer
client-go/informers/externalversions/api/v1/inferencepool.go - Import informer
client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go - Informer factory
client-go/informers/externalversions/factory.go - Import lister
client-go/listers/apix/v1alpha1/inferencepoolimport.go
Conformance
- Conformance suite
conformance/conformance.go - Served endpoint checks
conformance/utils/kubernetes/served_endpoint.go
Other components
- Inference client
- Envoy ext-proc
- Model servers
- Kubernetes API
Connections 22
- Inference client → Envoy ext-proc: sends inference
- Envoy ext-proc → Streaming handler: streams ext-proc
- Streaming handler → Endpoint picker: selects endpoint
pkg/lwepp/handlers/server.go - Streaming handler → Endpoint datastore: reads endpoints
pkg/lwepp/handlers/server.go - Endpoint picker → Model servers: routes request
- LWEPP process → Endpoint datastore: creates datastore
cmd/lwepp/main.go - LWEPP process → Controller manager: creates manager
cmd/lwepp/main.go - LWEPP process → Pool reconciler: sets up controllers
cmd/lwepp/main.go - Controller manager → InferencePool API: registers scheme
pkg/lwepp/server/controller_manager.go - Pool reconciler → InferencePool API: reconciles resource
pkg/lwepp/controller/inferencepool_reconciler.go - Pool reconciler → Endpoint datastore: updates state
pkg/lwepp/controller/inferencepool_reconciler.go - Pool reconciler → Kubernetes API: watches resources
- Pod reconciler → Endpoint datastore: updates state
- Versioned clientset → InferencePool API: types resources
client-go/clientset/versioned/clientset.go - Versioned clientset → Pool import API: types resources
client-go/clientset/versioned/clientset.go - Informer factory → Pool informer: provides informer
client-go/informers/externalversions/factory.go - Informer factory → Import informer: provides informer
client-go/informers/externalversions/factory.go - Pool informer → Versioned clientset: uses client
client-go/informers/externalversions/api/v1/inferencepool.go - Import informer → Versioned clientset: uses client
client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go - Import informer → Import lister: indexes resources
client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go - Conformance suite → Served endpoint checks: checks endpoint
conformance/conformance.go - Conformance suite → InferencePool API: tests API behavior
conformance/conformance.go