# kubernetes-sigs/gateway-api-inference-extension architecture

> Architecture diagram of the GitHub repository kubernetes-sigs/gateway-api-inference-extension, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kubernetes-sigs/gateway-api-inference-extension
- Repository: https://github.com/kubernetes-sigs/gateway-api-inference-extension

## Overview

This repository maintains the InferencePool Kubernetes API, a lightweight Envoy external-processing endpoint picker (LWEPP), generated Go client support, and conformance tests. The main runtime path is an inference request arriving through Envoy’s ext-proc integration, being handled by LWEPP and selecting a backend endpoint; separately, LWEPP watches pool and pod state through Kubernetes controllers and a datastore. The README documents the broader inference-gateway role and conformance scope. Some controller, datastore, and request-handling internals were not sampled, so those links are omitted where the available evidence does not establish them.

Main entry points: [`pkg/lwepp/controller/inferencepool_reconciler.go`](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/controller/inferencepool_reconciler.go) (Pool reconciler), [`client-go/clientset/versioned/clientset.go`](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/clientset/versioned/clientset.go) (Versioned clientset), [`pkg/lwepp/datastore/datastore.go`](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/datastore/datastore.go) (Endpoint datastore), [`api/v1/inferencepool_types.go`](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/api/v1/inferencepool_types.go) (InferencePool API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["LWEPP runtime"]
  node_lwepp["LWEPP process<br/>[main.go]"]
  node_manager["Controller manager"]
  node_stream["Streaming handler<br/>[server.go]"]
  node_picker["Endpoint picker<br/>[server.go]"]
  node_datastore[("Endpoint datastore<br/>[datastore.go]")]
  node_poolcontroller["Pool reconciler"]
  node_podcontroller["Pod reconciler<br/>[pod_reconciler.go]"]
end

subgraph group_kubeapi["Kubernetes APIs"]
  node_poolapi["InferencePool API"]
  node_importapi["Pool import API"]
end

subgraph group_clientgo["Go client support"]
  node_clientset["Versioned clientset<br/>[clientset.go]"]
  node_poolinformer["Pool informer<br/>[inferencepool.go]"]
  node_importinformer["Import informer"]
  node_informerfactory["Informer factory<br/>[factory.go]"]
  node_poollister["Import lister"]
end

subgraph group_conformance["Conformance"]
  node_conformance["Conformance suite<br/>[conformance.go]"]
  node_servedendpoint["Served endpoint checks<br/>[served_endpoint.go]"]
end

node_client(("Inference client"))
node_envoy["Envoy ext-proc"]
node_modelservers["Model servers"]
node_kube["Kubernetes API"]

node_client -->|"sends inference"| node_envoy
node_envoy -->|"streams ext-proc"| node_stream
node_stream -->|"selects endpoint"| node_picker
node_stream -->|"reads endpoints"| node_datastore
node_picker -->|"routes request"| node_modelservers
node_lwepp -->|"creates datastore"| node_datastore
node_lwepp -->|"creates manager"| node_manager
node_lwepp -->|"sets up controllers"| node_poolcontroller
node_manager -->|"registers scheme"| node_poolapi
node_poolcontroller -->|"reconciles resource"| node_poolapi
node_poolcontroller -->|"updates state"| node_datastore
node_poolcontroller -.->|"watches resources"| node_kube
node_podcontroller -.->|"updates state"| node_datastore
node_clientset -->|"types resources"| node_poolapi
node_clientset -->|"types resources"| node_importapi
node_informerfactory -->|"provides informer"| node_poolinformer
node_informerfactory -->|"provides informer"| node_importinformer
node_poolinformer -->|"uses client"| node_clientset
node_importinformer -->|"uses client"| node_clientset
node_importinformer -->|"indexes resources"| node_poollister
node_conformance -->|"checks endpoint"| node_servedendpoint
node_conformance -->|"tests API behavior"| node_poolapi

click node_lwepp "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/cmd/lwepp/main.go"
click node_manager "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/server/controller_manager.go"
click node_stream "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/handlers/server.go"
click node_picker "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/handlers/server.go"
click node_datastore "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/datastore/datastore.go"
click node_poolcontroller "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/controller/inferencepool_reconciler.go"
click node_podcontroller "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/controller/pod_reconciler.go"
click node_poolapi "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/api/v1/inferencepool_types.go"
click node_importapi "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/apix/v1alpha1/inferencepoolimport_types.go"
click node_clientset "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/clientset/versioned/clientset.go"
click node_poolinformer "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/informers/externalversions/api/v1/inferencepool.go"
click node_importinformer "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go"
click node_informerfactory "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/informers/externalversions/factory.go"
click node_poollister "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/listers/apix/v1alpha1/inferencepoolimport.go"
click node_conformance "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/conformance/conformance.go"
click node_servedendpoint "https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/conformance/utils/kubernetes/served_endpoint.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_lwepp,node_manager,node_stream,node_picker,node_datastore,node_poolcontroller,node_podcontroller,node_client toneBlue
class node_poolapi,node_importapi toneAmber
class node_clientset,node_poolinformer,node_importinformer,node_informerfactory,node_poollister,node_modelservers,node_kube toneMint
class node_conformance,node_servedendpoint toneRose
class node_envoy toneIndigo
```

## Components

### LWEPP runtime

- [LWEPP process](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/cmd/lwepp/main.go): `cmd/lwepp/main.go`
- [Controller manager](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/server/controller_manager.go): `pkg/lwepp/server/controller_manager.go`
- [Streaming handler](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/handlers/server.go): `pkg/lwepp/handlers/server.go`
- [Endpoint picker](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/handlers/server.go): `pkg/lwepp/handlers/server.go`
- [Endpoint datastore](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/datastore/datastore.go): `pkg/lwepp/datastore/datastore.go`
- [Pool reconciler](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/controller/inferencepool_reconciler.go): `pkg/lwepp/controller/inferencepool_reconciler.go`
- [Pod reconciler](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/pkg/lwepp/controller/pod_reconciler.go): `pkg/lwepp/controller/pod_reconciler.go`

### Kubernetes APIs

- [InferencePool API](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/api/v1/inferencepool_types.go): `api/v1/inferencepool_types.go`
- [Pool import API](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/apix/v1alpha1/inferencepoolimport_types.go): `apix/v1alpha1/inferencepoolimport_types.go`

### Go client support

- [Versioned clientset](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/clientset/versioned/clientset.go): `client-go/clientset/versioned/clientset.go`
- [Pool informer](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/informers/externalversions/api/v1/inferencepool.go): `client-go/informers/externalversions/api/v1/inferencepool.go`
- [Import informer](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go): `client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go`
- [Informer factory](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/informers/externalversions/factory.go): `client-go/informers/externalversions/factory.go`
- [Import lister](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/client-go/listers/apix/v1alpha1/inferencepoolimport.go): `client-go/listers/apix/v1alpha1/inferencepoolimport.go`

### Conformance

- [Conformance suite](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/conformance/conformance.go): `conformance/conformance.go`
- [Served endpoint checks](https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/main/conformance/utils/kubernetes/served_endpoint.go): `conformance/utils/kubernetes/served_endpoint.go`

### Other components

- **Inference client**
- **Envoy ext-proc**
- **Model servers**
- **Kubernetes API**

## Connections

- Inference client → Envoy ext-proc: sends inference
- Envoy ext-proc → Streaming handler: streams ext-proc
- Streaming handler → Endpoint picker: selects endpoint (evidence: `pkg/lwepp/handlers/server.go`)
- Streaming handler → Endpoint datastore: reads endpoints (evidence: `pkg/lwepp/handlers/server.go`)
- Endpoint picker → Model servers: routes request
- LWEPP process → Endpoint datastore: creates datastore (evidence: `cmd/lwepp/main.go`)
- LWEPP process → Controller manager: creates manager (evidence: `cmd/lwepp/main.go`)
- LWEPP process → Pool reconciler: sets up controllers (evidence: `cmd/lwepp/main.go`)
- Controller manager → InferencePool API: registers scheme (evidence: `pkg/lwepp/server/controller_manager.go`)
- Pool reconciler → InferencePool API: reconciles resource (evidence: `pkg/lwepp/controller/inferencepool_reconciler.go`)
- Pool reconciler → Endpoint datastore: updates state (evidence: `pkg/lwepp/controller/inferencepool_reconciler.go`)
- Pool reconciler → Kubernetes API: watches resources
- Pod reconciler → Endpoint datastore: updates state
- Versioned clientset → InferencePool API: types resources (evidence: `client-go/clientset/versioned/clientset.go`)
- Versioned clientset → Pool import API: types resources (evidence: `client-go/clientset/versioned/clientset.go`)
- Informer factory → Pool informer: provides informer (evidence: `client-go/informers/externalversions/factory.go`)
- Informer factory → Import informer: provides informer (evidence: `client-go/informers/externalversions/factory.go`)
- Pool informer → Versioned clientset: uses client (evidence: `client-go/informers/externalversions/api/v1/inferencepool.go`)
- Import informer → Versioned clientset: uses client (evidence: `client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go`)
- Import informer → Import lister: indexes resources (evidence: `client-go/informers/externalversions/apix/v1alpha1/inferencepoolimport.go`)
- Conformance suite → Served endpoint checks: checks endpoint (evidence: `conformance/conformance.go`)
- Conformance suite → InferencePool API: tests API behavior (evidence: `conformance/conformance.go`)
