How aryateja2106/nvidia-dynamo works
Project Overview and Purpose
• This is a high-throughput, low-latency distributed inference framework for generative AI and reasoning models. Its goal is to serve large language models (LLMs) using multiple backends (e.g., TRT-LLM, vLLM, SGLang) while efficiently managing GPU resources, routing requests, and handling KV cache offloading.
• The system is built in Rust for performance-critical components and in Python for extensibility. It supports multi-node environments, with key areas like LLM serving components and a distributed runtime.
File Structure to Identify Main Components
• Components Directory: Contains modules such as “components/http” (likely the OpenAI-compatible frontend server) and “components/metrics” (for instrumentation). These represent core service elements that directly handle request processing and system observation.
• Container & Deploy Folders: There are Dockerfiles, Kubernetes manifests (in deploy/Kubernetes and Helm charts) and docker-compose files. These files indicate containerization, orchestration, and the ability to deploy the platform in a distributed fashion.
• Deploy/dynamo: Encompasses the API server, operator, and database management. The operator specifically ties into Kubernetes CRDs and controllers, suggesting that the system uses an operator framework for lifecycle management.
• Launch Folder & Examples: Provide runnable samples and demo scripts (e.g., dynamo-run, llmctl) that illustrate how to start and interact with the system.
Main Components
a. Inference Serving & Frontends:
• OpenAI-Compatible HTTP API Server (components/http and lib/llm’s openai protocols)
• CLI Tools (from launch/ and examples/ directories) that allow local testing and serving.
b. Routing and Scheduling:
• Basic Router components and KV-aware routers (found in examples/llm/components and in lib/llm, particularly kv_router modules)
• Dynamic GPU and request scheduling logic
c. Worker / Backend Engines:
• Different backend engines (TRT-LLM, vLLM, SGLang, MistralRS) integrated in lib/llm/engines and container subdirectories.
• Worker modules responsible for inference computation.
d. Distributed Runtime and Orchestration:
• Distributed runtime services orchestrated via Docker Compose (deploy/docker-compose.yml) or Kubernetes deployments (deploy/Kubernetes, deploy/dynamo/operator).
• The Dynamo Operator handling CRDs and the deployment lifecycle.
• Additional deployment using Helm charts.
e. Monitoring and Metrics:
• Metrics components (in both components/metrics and deploy/metrics) that include Prometheus and Grafana dashboards.
f. SDK / Bindings:
• Language bindings (in lib/bindings) for Python, C, and C++ which facilitate integration with external systems.
Relationships and Data Flow
• How client requests (via REST API endpoints exposed by the frontend HTTP server) flow through the router.
• That requests are forwarded from the router to specialized backend workers (each one potentially running a different LLM engine).
• Connections to the KV caching layer; the design eliminates redundant KV cache recomputation.
• Communication with orchestration components (e.g., the operator managing deployments, the underlying message queues using protocols such as NATS, TCP, or even ZeroMQ as indicated by file names in lib/runtime).
Notable Architectural Patterns and Technologies
• Microservices/Modular design: Separating HTTP frontends, routers, and backend workers, each with clear responsibilities.
• Distributed system design: Focus on multi-node, containerized deployment using Docker, Kubernetes, and Helm.
• Polyglot approaches: Use of Rust for core performance, Python for extensibility, and C/C++ for language bindings.
• Dynamic request routing and KV cache management as key performance optimizations.
• Observability: Integrated metrics and logging as found in components/metrics and lib/runtime/logging, along with dashboards in deploy/metrics.