How substratusai/kubeai works
KubeAI is a Kubernetes inference operator and OpenAI-compatible proxy. A client request enters the HTTP server, reaches the proxy, which scales a model and selects a backend before forwarding the request; the operator reconciles Model resources into serving Pods. Separate branches cover event-stream requests, autoscaling, and model-engine provisioning. The README documents product capabilities such as model caching and dynamic adapters, but sampled excerpts do not expose all their runtime wiring, so some links are shown as unverified.
Main entry points: model_controller.go Model reconciler · modelproxy/handler.go Request proxy · messenger.go Event messenger · openaiserver/handler.go OpenAI HTTP server
Components 22
Inference API
- OpenAI HTTP server
internal/openaiserver/handler.go - Request proxy
internal/modelproxy/handler.go - Request parsing
internal/modelproxy/request.go - Model listing
internal/openaiserver/models.go - Event messenger
internal/messenger/messenger.go
Routing and scaling
- Model lookup client
internal/modelclient/client.go - Backend load balancer
internal/loadbalancer/load_balancer.go - Prefix-aware routing
internal/loadbalancer/balance_chwbl.go - Model autoscaler
internal/modelautoscaler/autoscaler.go
Model operator
- Model resource
api/k8s/v1/model_types.go - Model reconciler
internal/modelcontroller/model_controller.go - Model cache and adapters
internal/modelcontroller/cache.go - Model sources
internal/modelcontroller/model_source.go - vLLM Pod builder
internal/modelcontroller/engine_vllm.go - Ollama Pod builder
internal/modelcontroller/engine_ollama.go - FasterWhisper Pod builder
internal/modelcontroller/engine_fasterwhisper.go - Infinity Pod builder
internal/modelcontroller/engine_infinity.go
Serving runtime
- Model serving Pods
- Inference engines
Other components
- Inference client
- Pub/Sub service
- Kubernetes API
Connections 23
- Inference client → OpenAI HTTP server: sends request
- OpenAI HTTP server → Request proxy: routes inference
internal/openaiserver/handler.go - OpenAI HTTP server → Model listing: routes model listing
internal/openaiserver/handler.go - Request proxy → Request parsing: parses request
internal/modelproxy/handler.go - Request proxy → Model lookup client: looks up and scales
- Request proxy → Backend load balancer: selects backend
- Backend load balancer → Prefix-aware routing: uses routing strategy
- Request proxy → Model serving Pods: forwards inference
- Model serving Pods → Inference engines: runs model engine
- Event messenger → Model lookup client: looks up and scales
internal/messenger/messenger.go - Event messenger → Backend load balancer: selects backend
internal/messenger/messenger.go - Pub/Sub service → Event messenger: delivers requests
- Event messenger → Pub/Sub service: publishes responses
internal/messenger/messenger.go - Model autoscaler → Model lookup client: lists and scales
internal/modelautoscaler/autoscaler.go - Model resource → Model reconciler: reconciles model
- Model reconciler → Kubernetes API: reads and updates
internal/modelcontroller/model_controller.go - Model reconciler → Model sources: resolves source
- Model reconciler → Model cache and adapters: configures cache and adapters
- Model reconciler → vLLM Pod builder: builds serving Pod
- Model reconciler → Ollama Pod builder: builds serving Pod
- Model reconciler → FasterWhisper Pod builder: builds serving Pod
- Model reconciler → Infinity Pod builder: builds serving Pod
- Kubernetes API → Model serving Pods: schedules Pods