# ai-dynamo/dynamo architecture

> Architecture diagram of the GitHub repository ai-dynamo/dynamo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ai-dynamo/dynamo
- Repository: https://github.com/ai-dynamo/dynamo
- Video tour (about a minute): https://gitdiagram.com/ai-dynamo/dynamo/video

## Overview

Dynamo is a datacenter-scale inference orchestration layer for coordinating inference engines across GPUs and nodes. Its principal workflow accepts inference requests, routes them to engine workers, and returns generated output; documented capabilities add KV-aware routing, disaggregated prefill/decode, and SLA-driven autoscaling. The graph also includes the engine integration and performance-model layers. Several subsystem links reflect README-level capability descriptions rather than inspected call sites, so only documented relationships are shown.

Main entry points: [`components/src/dynamo/router/args.py`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/router/args.py) (Request router), [`lib/backend-common/src/engine.rs`](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/engine.rs) (Backend engines), [`components/src/dynamo/sglang/request_handlers/llm/decode_handler.py`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/sglang/request_handlers/llm/decode_handler.py) (Decode worker), [`components/src/dynamo/planner/core/state_machine.py`](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/planner/core/state_machine.py) (SLA planner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Request serving"]
  node_frontend["Request frontend<br/>[main.py]"]
  node_router["Request router<br/>[args.py]"]
  node_global_router["Global routing<br/>[handler.py]"]
  node_prefill["Prefill worker<br/>[prefill_handler.py]"]
  node_decode["Decode worker<br/>[decode_handler.py]"]
  node_response["Inference response"]
end

subgraph group_engines["Engine integration"]
  node_engines["Backend engines<br/>[engine.rs]"]
  node_sglang["SGLang adapter<br/>[engine.rs]"]
  node_vllm["vLLM adapter<br/>[engine.rs]"]
  node_trtllm["TensorRT-LLM adapter<br/>[engine.rs]"]
end

subgraph group_planning["Planning and scaling"]
  node_planner["SLA planner<br/>[state_machine.py]"]
  node_perf_model["Performance estimates<br/>[engine_query.py]"]
  node_global_planner["Global scaling<br/>[orchestrator.py]"]
end

subgraph group_kv["KV coordination"]
  node_kv_router["KV-aware routing<br/>[worker_query.rs]"]
  node_kv_storage[("KV storage<br/>[storage.rs]")]
  node_offload["KV offload pipeline<br/>[pipeline.rs]"]
end

subgraph group_runtime["Runtime contracts"]
  node_python_engine["Python engine bridge<br/>[engine.rs]"]
  node_runtime_engine["Runtime engine<br/>[engine.rs]"]
end

node_client(("Inference client"))

node_client -->|"submits request"| node_frontend
node_frontend -->|"routes request"| node_router
node_router -.->|"selects backend"| node_global_router
node_router -->|"dispatches prefill"| node_prefill
node_router -->|"dispatches decode"| node_decode
node_prefill -->|"transfers KV state"| node_decode
node_prefill -->|"uses engine contract"| node_engines
node_decode -->|"uses engine contract"| node_engines
node_engines -->|"adapts SGLang"| node_sglang
node_engines -->|"adapts vLLM"| node_vllm
node_engines -->|"adapts TensorRT-LLM"| node_trtllm
node_decode -->|"returns output"| node_response
node_router -->|"uses KV hints"| node_kv_router
node_kv_router -.->|"consults cache state"| node_kv_storage
node_kv_storage -.->|"offloads blocks"| node_offload
node_planner -->|"estimates capacity"| node_perf_model
node_planner -.->|"coordinates scaling"| node_global_planner
node_planner -->|"scales serving pools"| node_router
node_python_engine -->|"bridges engine calls"| node_runtime_engine

click node_frontend "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/frontend/main.py"
click node_router "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/router/args.py"
click node_global_router "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/global_router/handler.py"
click node_prefill "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/sglang/request_handlers/llm/prefill_handler.py"
click node_decode "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/sglang/request_handlers/llm/decode_handler.py"
click node_engines "https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/engine.rs"
click node_sglang "https://github.com/ai-dynamo/dynamo/blob/main/lib/sidecar/sglang/src/engine.rs"
click node_vllm "https://github.com/ai-dynamo/dynamo/blob/main/lib/sidecar/vllm/src/engine.rs"
click node_trtllm "https://github.com/ai-dynamo/dynamo/blob/main/lib/sidecar/trtllm/src/engine.rs"
click node_planner "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/planner/core/state_machine.py"
click node_perf_model "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/planner/core/perf_model/engine_query.py"
click node_global_planner "https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/global_planner/orchestrator.py"
click node_kv_router "https://github.com/ai-dynamo/dynamo/blob/main/lib/llm/src/kv_router/indexer/recovery/worker_query.rs"
click node_kv_storage "https://github.com/ai-dynamo/dynamo/blob/main/lib/llm/src/kv/storage.rs"
click node_offload "https://github.com/ai-dynamo/dynamo/blob/main/lib/kvbm-engine/src/offload/pipeline.rs"
click node_python_engine "https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/rust/engine.rs"
click node_runtime_engine "https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/engine.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend,node_router,node_global_router,node_prefill,node_decode,node_response,node_client toneBlue
class node_engines,node_sglang,node_vllm,node_trtllm toneAmber
class node_planner,node_perf_model,node_global_planner toneMint
class node_kv_router,node_kv_storage,node_offload toneRose
class node_python_engine,node_runtime_engine toneIndigo
```

## Components

### Request serving

- [Request frontend](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/frontend/main.py): `components/src/dynamo/frontend/main.py`
- [Request router](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/router/args.py): `components/src/dynamo/router/args.py`
- [Global routing](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/global_router/handler.py): `components/src/dynamo/global_router/handler.py`
- [Prefill worker](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/sglang/request_handlers/llm/prefill_handler.py): `components/src/dynamo/sglang/request_handlers/llm/prefill_handler.py`
- [Decode worker](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/sglang/request_handlers/llm/decode_handler.py): `components/src/dynamo/sglang/request_handlers/llm/decode_handler.py`
- **Inference response**

### Engine integration

- [Backend engines](https://github.com/ai-dynamo/dynamo/blob/main/lib/backend-common/src/engine.rs): `lib/backend-common/src/engine.rs`
- [SGLang adapter](https://github.com/ai-dynamo/dynamo/blob/main/lib/sidecar/sglang/src/engine.rs): `lib/sidecar/sglang/src/engine.rs`
- [vLLM adapter](https://github.com/ai-dynamo/dynamo/blob/main/lib/sidecar/vllm/src/engine.rs): `lib/sidecar/vllm/src/engine.rs`
- [TensorRT-LLM adapter](https://github.com/ai-dynamo/dynamo/blob/main/lib/sidecar/trtllm/src/engine.rs): `lib/sidecar/trtllm/src/engine.rs`

### Planning and scaling

- [SLA planner](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/planner/core/state_machine.py): `components/src/dynamo/planner/core/state_machine.py`
- [Performance estimates](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/planner/core/perf_model/engine_query.py): `components/src/dynamo/planner/core/perf_model/engine_query.py`
- [Global scaling](https://github.com/ai-dynamo/dynamo/blob/main/components/src/dynamo/global_planner/orchestrator.py): `components/src/dynamo/global_planner/orchestrator.py`

### KV coordination

- [KV-aware routing](https://github.com/ai-dynamo/dynamo/blob/main/lib/llm/src/kv_router/indexer/recovery/worker_query.rs): `lib/llm/src/kv_router/indexer/recovery/worker_query.rs`
- [KV storage](https://github.com/ai-dynamo/dynamo/blob/main/lib/llm/src/kv/storage.rs): `lib/llm/src/kv/storage.rs`
- [KV offload pipeline](https://github.com/ai-dynamo/dynamo/blob/main/lib/kvbm-engine/src/offload/pipeline.rs): `lib/kvbm-engine/src/offload/pipeline.rs`

### Runtime contracts

- [Python engine bridge](https://github.com/ai-dynamo/dynamo/blob/main/lib/bindings/python/rust/engine.rs): `lib/bindings/python/rust/engine.rs`
- [Runtime engine](https://github.com/ai-dynamo/dynamo/blob/main/lib/runtime/src/engine.rs): `lib/runtime/src/engine.rs`

### Other components

- **Inference client**

## Connections

- Inference client → Request frontend: submits request (evidence: `README.md`)
- Request frontend → Request router: routes request (evidence: `README.md`)
- Request router → Global routing: selects backend
- Request router → Prefill worker: dispatches prefill (evidence: `README.md`)
- Request router → Decode worker: dispatches decode (evidence: `README.md`)
- Prefill worker → Decode worker: transfers KV state (evidence: `README.md`)
- Prefill worker → Backend engines: uses engine contract (evidence: `lib/backend-common/src/engine.rs`)
- Decode worker → Backend engines: uses engine contract (evidence: `lib/backend-common/src/engine.rs`)
- Backend engines → SGLang adapter: adapts SGLang (evidence: `lib/sidecar/sglang/src/engine.rs`)
- Backend engines → vLLM adapter: adapts vLLM (evidence: `lib/sidecar/vllm/src/engine.rs`)
- Backend engines → TensorRT-LLM adapter: adapts TensorRT-LLM (evidence: `lib/sidecar/trtllm/src/engine.rs`)
- Decode worker → Inference response: returns output (evidence: `README.md`)
- Request router → KV-aware routing: uses KV hints (evidence: `README.md`)
- KV-aware routing → KV storage: consults cache state
- KV storage → KV offload pipeline: offloads blocks
- SLA planner → Performance estimates: estimates capacity (evidence: `components/src/dynamo/planner/core/perf_model/engine_query.py`)
- SLA planner → Global scaling: coordinates scaling
- SLA planner → Request router: scales serving pools (evidence: `README.md`)
- Python engine bridge → Runtime engine: bridges engine calls (evidence: `lib/bindings/python/rust/engine.rs`)
