How ai-dynamo/dynamo works
Dynamo is a datacenter-scale inference orchestration layer for coordinating inference engines across GPUs and nodes. Its principal workflow accepts inference requests, routes them to engine workers, and returns generated output; documented capabilities add KV-aware routing, disaggregated prefill/decode, and SLA-driven autoscaling. The graph also includes the engine integration and performance-model layers. Several subsystem links reflect README-level capability descriptions rather than inspected call sites, so only documented relationships are shown.
Main entry points: args.py Request router · engine.rs Backend engines · decode_handler.py Decode worker · state_machine.py SLA planner
Components 19
Request serving
- Request frontend
components/src/dynamo/frontend/main.py - Request router
components/src/dynamo/router/args.py - Global routing
components/src/dynamo/global_router/handler.py - Prefill worker
components/src/dynamo/sglang/request_handlers/llm/prefill_handler.py - Decode worker
components/src/dynamo/sglang/request_handlers/llm/decode_handler.py - Inference response
Engine integration
- Backend engines
lib/backend-common/src/engine.rs - SGLang adapter
lib/sidecar/sglang/src/engine.rs - vLLM adapter
lib/sidecar/vllm/src/engine.rs - TensorRT-LLM adapter
lib/sidecar/trtllm/src/engine.rs
Planning and scaling
- SLA planner
components/src/dynamo/planner/core/state_machine.py - Performance estimates
components/src/dynamo/planner/core/perf_model/engine_query.py - Global scaling
components/src/dynamo/global_planner/orchestrator.py
KV coordination
- KV-aware routing
lib/llm/src/kv_router/indexer/recovery/worker_query.rs - KV storage
lib/llm/src/kv/storage.rs - KV offload pipeline
lib/kvbm-engine/src/offload/pipeline.rs
Runtime contracts
- Python engine bridge
lib/bindings/python/rust/engine.rs - Runtime engine
lib/runtime/src/engine.rs
Other components
- Inference client
Connections 19
- Inference client → Request frontend: submits request
README.md - Request frontend → Request router: routes request
README.md - Request router → Global routing: selects backend
- Request router → Prefill worker: dispatches prefill
README.md - Request router → Decode worker: dispatches decode
README.md - Prefill worker → Decode worker: transfers KV state
README.md - Prefill worker → Backend engines: uses engine contract
lib/backend-common/src/engine.rs - Decode worker → Backend engines: uses engine contract
lib/backend-common/src/engine.rs - Backend engines → SGLang adapter: adapts SGLang
lib/sidecar/sglang/src/engine.rs - Backend engines → vLLM adapter: adapts vLLM
lib/sidecar/vllm/src/engine.rs - Backend engines → TensorRT-LLM adapter: adapts TensorRT-LLM
lib/sidecar/trtllm/src/engine.rs - Decode worker → Inference response: returns output
README.md - Request router → KV-aware routing: uses KV hints
README.md - KV-aware routing → KV storage: consults cache state
- KV storage → KV offload pipeline: offloads blocks
- SLA planner → Performance estimates: estimates capacity
components/src/dynamo/planner/core/perf_model/engine_query.py - SLA planner → Global scaling: coordinates scaling
- SLA planner → Request router: scales serving pools
README.md - Python engine bridge → Runtime engine: bridges engine calls
lib/bindings/python/rust/engine.rs