# vllm-project/aibrix architecture

> Architecture diagram of the GitHub repository vllm-project/aibrix, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vllm-project/aibrix
- Repository: https://github.com/vllm-project/aibrix

## Overview

AIBrix is cloud-native infrastructure for deploying, routing, and scaling LLM inference. A principal operator workflow is to submit a deployment through the console, have the control plane plan and provision resources, and expose a running model for inference. The runtime also includes autoscaling, model-adapter management, standardized engine metrics, batch dispatch, and distributed KV-cache capabilities. The graph combines README-documented capabilities with the sampled controller, batch, metrics, and runtime source; several broader subsystems are represented at their existing directory paths because their implementation was not sampled.

Main entry points: [`python/aibrix/aibrix/runtime`](https://github.com/vllm-project/aibrix/tree/main/python/aibrix/aibrix/runtime) (Model Engines), [`apps/console/api/handler`](https://github.com/vllm-project/aibrix/tree/main/apps/console/api/handler) (Console API), [`apps/console/web/src`](https://github.com/vllm-project/aibrix/tree/main/apps/console/web/src) (Console UI), [`apps/console/api/planner/impl`](https://github.com/vllm-project/aibrix/tree/main/apps/console/api/planner/impl) (Job Planner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_console["Management Console"]
  node_console_web["Console UI"]
  node_console_api["Console API"]
  node_planner["Job Planner"]
  node_resource_manager["Resource Manager"]
  node_resource_providers["Cloud Providers"]
  node_console_store[("Console Store")]
end

subgraph group_control["Kubernetes Control"]
  node_api_types["Custom Resources"]
  node_model_adapter["LoRA Controller"]
  node_pod_autoscaler["Pod Autoscaler"]
end

subgraph group_serving["Inference Serving"]
  node_gateway["LLM Gateway"]
  node_engine["Model Engines"]
  node_batch_dispatch["Batch Dispatch<br/>[engine.py]"]
  node_batch_storage[("Batch Storage<br/>[batch_storage.py]")]
end

subgraph group_cache["KV Cache"]
  node_kv_manager["KV Cache Manager<br/>[cache_manager.py]"]
  node_kv_service["Distributed KV Cache"]
end

subgraph group_observe["Metrics and Scaling"]
  node_metrics_fetcher["Engine Metrics<br/>[engine_fetcher.go]"]
  node_metric_rules["Metric Standardization<br/>[engine_rules.py]"]
  node_autoscaling_api["Autoscaling API"]
end

node_operator(("Platform Operator"))
node_kubernetes(("Kubernetes"))

node_operator -->|"submits deployment"| node_console_web
node_console_web -->|"calls API"| node_console_api
node_console_api -->|"persists records"| node_console_store
node_console_api -->|"submits jobs"| node_planner
node_planner -->|"requests resources"| node_resource_manager
node_resource_manager -->|"selects provider"| node_resource_providers
node_resource_providers -.->|"provisions workloads"| node_kubernetes
node_kubernetes -->|"stores resources"| node_api_types
node_kubernetes -->|"reconciles adapters"| node_model_adapter
node_kubernetes -->|"reconciles scaling"| node_pod_autoscaler
node_operator -->|"sends inference"| node_gateway
node_gateway -->|"routes requests"| node_engine
node_operator -->|"supplies batch input"| node_batch_storage
node_batch_storage -->|"reads requests"| node_batch_dispatch
node_batch_dispatch -->|"dispatches work"| node_engine
node_engine -.->|"uses cache"| node_kv_manager
node_kv_manager -->|"reads and writes"| node_kv_service
node_engine -->|"exposes metrics"| node_metrics_fetcher
node_metrics_fetcher -.->|"standardizes metrics"| node_metric_rules
node_autoscaling_api -->|"configures scaling"| node_pod_autoscaler

click node_console_web "https://github.com/vllm-project/aibrix/tree/main/apps/console/web/src"
click node_console_api "https://github.com/vllm-project/aibrix/tree/main/apps/console/api/handler"
click node_planner "https://github.com/vllm-project/aibrix/tree/main/apps/console/api/planner/impl"
click node_resource_manager "https://github.com/vllm-project/aibrix/tree/main/apps/console/api/resource_manager"
click node_resource_providers "https://github.com/vllm-project/aibrix/tree/main/apps/console/api/resource_manager/provider"
click node_console_store "https://github.com/vllm-project/aibrix/tree/main/apps/console/api/store"
click node_api_types "https://github.com/vllm-project/aibrix/tree/main/api"
click node_model_adapter "https://github.com/vllm-project/aibrix/blob/main/pkg/controller/modeladapter/modeladapter_controller.go"
click node_pod_autoscaler "https://github.com/vllm-project/aibrix/blob/main/pkg/controller/podautoscaler/podautoscaler_controller.go"
click node_gateway "https://github.com/vllm-project/aibrix/tree/main/python/aibrix/aibrix/openai_frontend"
click node_engine "https://github.com/vllm-project/aibrix/tree/main/python/aibrix/aibrix/runtime"
click node_batch_dispatch "https://github.com/vllm-project/aibrix/blob/main/python/aibrix/aibrix/batch/client/engine.py"
click node_batch_storage "https://github.com/vllm-project/aibrix/blob/main/python/aibrix/aibrix/batch/storage/batch_storage.py"
click node_kv_manager "https://github.com/vllm-project/aibrix/blob/main/python/aibrix_kvcache/aibrix_kvcache/cache_manager.py"
click node_kv_service "https://github.com/vllm-project/aibrix/tree/main/pkg/cache/kvcache"
click node_metrics_fetcher "https://github.com/vllm-project/aibrix/blob/main/pkg/metrics/engine_fetcher.go"
click node_metric_rules "https://github.com/vllm-project/aibrix/blob/main/python/aibrix/aibrix/metrics/engine_rules.py"
click node_autoscaling_api "https://github.com/vllm-project/aibrix/blob/main/api/autoscaling/v1alpha1/podautoscaler_types.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_console_web,node_console_api,node_planner,node_resource_manager,node_resource_providers,node_console_store toneBlue
class node_api_types,node_model_adapter,node_pod_autoscaler toneAmber
class node_gateway,node_engine,node_batch_dispatch,node_batch_storage toneMint
class node_kv_manager,node_kv_service toneRose
class node_metrics_fetcher,node_metric_rules,node_autoscaling_api,node_operator,node_kubernetes toneIndigo
```

## Components

### Management Console

- [Console UI](https://github.com/vllm-project/aibrix/tree/main/apps/console/web/src): `apps/console/web/src`
- [Console API](https://github.com/vllm-project/aibrix/tree/main/apps/console/api/handler): `apps/console/api/handler`
- [Job Planner](https://github.com/vllm-project/aibrix/tree/main/apps/console/api/planner/impl): `apps/console/api/planner/impl`
- [Resource Manager](https://github.com/vllm-project/aibrix/tree/main/apps/console/api/resource_manager): `apps/console/api/resource_manager`
- [Cloud Providers](https://github.com/vllm-project/aibrix/tree/main/apps/console/api/resource_manager/provider): `apps/console/api/resource_manager/provider`
- [Console Store](https://github.com/vllm-project/aibrix/tree/main/apps/console/api/store): `apps/console/api/store`

### Kubernetes Control

- [Custom Resources](https://github.com/vllm-project/aibrix/tree/main/api): `api`
- [LoRA Controller](https://github.com/vllm-project/aibrix/blob/main/pkg/controller/modeladapter/modeladapter_controller.go): `pkg/controller/modeladapter/modeladapter_controller.go`
- [Pod Autoscaler](https://github.com/vllm-project/aibrix/blob/main/pkg/controller/podautoscaler/podautoscaler_controller.go): `pkg/controller/podautoscaler/podautoscaler_controller.go`

### Inference Serving

- [LLM Gateway](https://github.com/vllm-project/aibrix/tree/main/python/aibrix/aibrix/openai_frontend): `python/aibrix/aibrix/openai_frontend`
- [Model Engines](https://github.com/vllm-project/aibrix/tree/main/python/aibrix/aibrix/runtime): `python/aibrix/aibrix/runtime`
- [Batch Dispatch](https://github.com/vllm-project/aibrix/blob/main/python/aibrix/aibrix/batch/client/engine.py): `python/aibrix/aibrix/batch/client/engine.py`
- [Batch Storage](https://github.com/vllm-project/aibrix/blob/main/python/aibrix/aibrix/batch/storage/batch_storage.py): `python/aibrix/aibrix/batch/storage/batch_storage.py`

### KV Cache

- [KV Cache Manager](https://github.com/vllm-project/aibrix/blob/main/python/aibrix_kvcache/aibrix_kvcache/cache_manager.py): `python/aibrix_kvcache/aibrix_kvcache/cache_manager.py`
- [Distributed KV Cache](https://github.com/vllm-project/aibrix/tree/main/pkg/cache/kvcache): `pkg/cache/kvcache`

### Metrics and Scaling

- [Engine Metrics](https://github.com/vllm-project/aibrix/blob/main/pkg/metrics/engine_fetcher.go): `pkg/metrics/engine_fetcher.go`
- [Metric Standardization](https://github.com/vllm-project/aibrix/blob/main/python/aibrix/aibrix/metrics/engine_rules.py): `python/aibrix/aibrix/metrics/engine_rules.py`
- [Autoscaling API](https://github.com/vllm-project/aibrix/blob/main/api/autoscaling/v1alpha1/podautoscaler_types.go): `api/autoscaling/v1alpha1/podautoscaler_types.go`

### Other components

- **Platform Operator**
- **Kubernetes**

## Connections

- Platform Operator → Console UI: submits deployment
- Console UI → Console API: calls API
- Console API → Console Store: persists records
- Console API → Job Planner: submits jobs
- Job Planner → Resource Manager: requests resources
- Resource Manager → Cloud Providers: selects provider
- Cloud Providers → Kubernetes: provisions workloads
- Kubernetes → Custom Resources: stores resources
- Kubernetes → LoRA Controller: reconciles adapters
- Kubernetes → Pod Autoscaler: reconciles scaling
- Platform Operator → LLM Gateway: sends inference
- LLM Gateway → Model Engines: routes requests
- Platform Operator → Batch Storage: supplies batch input
- Batch Storage → Batch Dispatch: reads requests
- Batch Dispatch → Model Engines: dispatches work
- Model Engines → KV Cache Manager: uses cache
- KV Cache Manager → Distributed KV Cache: reads and writes
- Model Engines → Engine Metrics: exposes metrics
- Engine Metrics → Metric Standardization: standardizes metrics
- Autoscaling API → Pod Autoscaler: configures scaling