# substratusai/kubeai architecture

> Architecture diagram of the GitHub repository substratusai/kubeai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/substratusai/kubeai
- Repository: https://github.com/substratusai/kubeai

## Overview

KubeAI is a Kubernetes inference operator and OpenAI-compatible proxy. A client request enters the HTTP server, reaches the proxy, which scales a model and selects a backend before forwarding the request; the operator reconciles Model resources into serving Pods. Separate branches cover event-stream requests, autoscaling, and model-engine provisioning. The README documents product capabilities such as model caching and dynamic adapters, but sampled excerpts do not expose all their runtime wiring, so some links are shown as unverified.

Main entry points: [`internal/modelcontroller/model_controller.go`](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/model_controller.go) (Model reconciler), [`internal/modelproxy/handler.go`](https://github.com/substratusai/kubeai/blob/main/internal/modelproxy/handler.go) (Request proxy), [`internal/messenger/messenger.go`](https://github.com/substratusai/kubeai/blob/main/internal/messenger/messenger.go) (Event messenger), [`internal/openaiserver/handler.go`](https://github.com/substratusai/kubeai/blob/main/internal/openaiserver/handler.go) (OpenAI HTTP server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Inference API"]
  node_http["OpenAI HTTP server<br/>[handler.go]"]
  node_proxy["Request proxy<br/>[handler.go]"]
  node_request["Request parsing<br/>[request.go]"]
  node_openai_models["Model listing<br/>[models.go]"]
  node_messenger["Event messenger<br/>[messenger.go]"]
end

subgraph group_routing["Routing and scaling"]
  node_model_client["Model lookup client<br/>[client.go]"]
  node_load_balancer["Backend load balancer<br/>[load_balancer.go]"]
  node_chwbl["Prefix-aware routing<br/>[balance_chwbl.go]"]
  node_autoscaler["Model autoscaler<br/>[autoscaler.go]"]
end

subgraph group_operator["Model operator"]
  node_model_crd["Model resource<br/>[model_types.go]"]
  node_controller["Model reconciler"]
  node_cache["Model cache and adapters<br/>[cache.go]"]
  node_model_source["Model sources<br/>[model_source.go]"]
  node_engine_vllm["vLLM Pod builder<br/>[engine_vllm.go]"]
  node_engine_ollama["Ollama Pod builder<br/>[engine_ollama.go]"]
  node_engine_whisper["FasterWhisper Pod builder"]
  node_engine_infinity["Infinity Pod builder<br/>[engine_infinity.go]"]
end

subgraph group_runtime["Serving runtime"]
  node_serving_pods["Model serving Pods"]
  node_engines["Inference engines"]
end

node_client(("Inference client"))
node_pubsub["Pub/Sub service"]
node_kubernetes["Kubernetes API"]

node_client -->|"sends request"| node_http
node_http -->|"routes inference"| node_proxy
node_http -->|"routes model listing"| node_openai_models
node_proxy -->|"parses request"| node_request
node_proxy -.->|"looks up and scales"| node_model_client
node_proxy -.->|"selects backend"| node_load_balancer
node_load_balancer -.->|"uses routing strategy"| node_chwbl
node_proxy -->|"forwards inference"| node_serving_pods
node_serving_pods -->|"runs model engine"| node_engines
node_messenger -->|"looks up and scales"| node_model_client
node_messenger -->|"selects backend"| node_load_balancer
node_pubsub -.->|"delivers requests"| node_messenger
node_messenger -->|"publishes responses"| node_pubsub
node_autoscaler -->|"lists and scales"| node_model_client
node_model_crd -.->|"reconciles model"| node_controller
node_controller -->|"reads and updates"| node_kubernetes
node_controller -.->|"resolves source"| node_model_source
node_controller -.->|"configures cache and adapters"| node_cache
node_controller -.->|"builds serving Pod"| node_engine_vllm
node_controller -.->|"builds serving Pod"| node_engine_ollama
node_controller -.->|"builds serving Pod"| node_engine_whisper
node_controller -.->|"builds serving Pod"| node_engine_infinity
node_kubernetes -->|"schedules Pods"| node_serving_pods

click node_http "https://github.com/substratusai/kubeai/blob/main/internal/openaiserver/handler.go"
click node_proxy "https://github.com/substratusai/kubeai/blob/main/internal/modelproxy/handler.go"
click node_request "https://github.com/substratusai/kubeai/blob/main/internal/modelproxy/request.go"
click node_openai_models "https://github.com/substratusai/kubeai/blob/main/internal/openaiserver/models.go"
click node_model_client "https://github.com/substratusai/kubeai/blob/main/internal/modelclient/client.go"
click node_load_balancer "https://github.com/substratusai/kubeai/blob/main/internal/loadbalancer/load_balancer.go"
click node_chwbl "https://github.com/substratusai/kubeai/blob/main/internal/loadbalancer/balance_chwbl.go"
click node_autoscaler "https://github.com/substratusai/kubeai/blob/main/internal/modelautoscaler/autoscaler.go"
click node_messenger "https://github.com/substratusai/kubeai/blob/main/internal/messenger/messenger.go"
click node_model_crd "https://github.com/substratusai/kubeai/blob/main/api/k8s/v1/model_types.go"
click node_controller "https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/model_controller.go"
click node_cache "https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/cache.go"
click node_model_source "https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/model_source.go"
click node_engine_vllm "https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_vllm.go"
click node_engine_ollama "https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_ollama.go"
click node_engine_whisper "https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_fasterwhisper.go"
click node_engine_infinity "https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_infinity.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_http,node_proxy,node_request,node_openai_models,node_messenger,node_client toneBlue
class node_model_client,node_load_balancer,node_chwbl,node_autoscaler toneAmber
class node_model_crd,node_controller,node_cache,node_model_source,node_engine_vllm,node_engine_ollama,node_engine_whisper,node_engine_infinity,node_kubernetes toneMint
class node_serving_pods,node_engines toneRose
class node_pubsub toneIndigo
```

## Components

### Inference API

- [OpenAI HTTP server](https://github.com/substratusai/kubeai/blob/main/internal/openaiserver/handler.go): `internal/openaiserver/handler.go`
- [Request proxy](https://github.com/substratusai/kubeai/blob/main/internal/modelproxy/handler.go): `internal/modelproxy/handler.go`
- [Request parsing](https://github.com/substratusai/kubeai/blob/main/internal/modelproxy/request.go): `internal/modelproxy/request.go`
- [Model listing](https://github.com/substratusai/kubeai/blob/main/internal/openaiserver/models.go): `internal/openaiserver/models.go`
- [Event messenger](https://github.com/substratusai/kubeai/blob/main/internal/messenger/messenger.go): `internal/messenger/messenger.go`

### Routing and scaling

- [Model lookup client](https://github.com/substratusai/kubeai/blob/main/internal/modelclient/client.go): `internal/modelclient/client.go`
- [Backend load balancer](https://github.com/substratusai/kubeai/blob/main/internal/loadbalancer/load_balancer.go): `internal/loadbalancer/load_balancer.go`
- [Prefix-aware routing](https://github.com/substratusai/kubeai/blob/main/internal/loadbalancer/balance_chwbl.go): `internal/loadbalancer/balance_chwbl.go`
- [Model autoscaler](https://github.com/substratusai/kubeai/blob/main/internal/modelautoscaler/autoscaler.go): `internal/modelautoscaler/autoscaler.go`

### Model operator

- [Model resource](https://github.com/substratusai/kubeai/blob/main/api/k8s/v1/model_types.go): `api/k8s/v1/model_types.go`
- [Model reconciler](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/model_controller.go): `internal/modelcontroller/model_controller.go`
- [Model cache and adapters](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/cache.go): `internal/modelcontroller/cache.go`
- [Model sources](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/model_source.go): `internal/modelcontroller/model_source.go`
- [vLLM Pod builder](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_vllm.go): `internal/modelcontroller/engine_vllm.go`
- [Ollama Pod builder](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_ollama.go): `internal/modelcontroller/engine_ollama.go`
- [FasterWhisper Pod builder](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_fasterwhisper.go): `internal/modelcontroller/engine_fasterwhisper.go`
- [Infinity Pod builder](https://github.com/substratusai/kubeai/blob/main/internal/modelcontroller/engine_infinity.go): `internal/modelcontroller/engine_infinity.go`

### Serving runtime

- **Model serving Pods**
- **Inference engines**

### Other components

- **Inference client**
- **Pub/Sub service**
- **Kubernetes API**

## Connections

- Inference client → OpenAI HTTP server: sends request
- OpenAI HTTP server → Request proxy: routes inference (evidence: `internal/openaiserver/handler.go`)
- OpenAI HTTP server → Model listing: routes model listing (evidence: `internal/openaiserver/handler.go`)
- Request proxy → Request parsing: parses request (evidence: `internal/modelproxy/handler.go`)
- Request proxy → Model lookup client: looks up and scales
- Request proxy → Backend load balancer: selects backend
- Backend load balancer → Prefix-aware routing: uses routing strategy
- Request proxy → Model serving Pods: forwards inference
- Model serving Pods → Inference engines: runs model engine
- Event messenger → Model lookup client: looks up and scales (evidence: `internal/messenger/messenger.go`)
- Event messenger → Backend load balancer: selects backend (evidence: `internal/messenger/messenger.go`)
- Pub/Sub service → Event messenger: delivers requests
- Event messenger → Pub/Sub service: publishes responses (evidence: `internal/messenger/messenger.go`)
- Model autoscaler → Model lookup client: lists and scales (evidence: `internal/modelautoscaler/autoscaler.go`)
- Model resource → Model reconciler: reconciles model
- Model reconciler → Kubernetes API: reads and updates (evidence: `internal/modelcontroller/model_controller.go`)
- Model reconciler → Model sources: resolves source
- Model reconciler → Model cache and adapters: configures cache and adapters
- Model reconciler → vLLM Pod builder: builds serving Pod
- Model reconciler → Ollama Pod builder: builds serving Pod
- Model reconciler → FasterWhisper Pod builder: builds serving Pod
- Model reconciler → Infinity Pod builder: builds serving Pod
- Kubernetes API → Model serving Pods: schedules Pods
