# aryateja2106/nvidia-dynamo architecture

> Architecture diagram of the GitHub repository aryateja2106/nvidia-dynamo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aryateja2106/nvidia-dynamo
- Repository: https://github.com/aryateja2106/nvidia-dynamo

## Overview

##### Project Overview and Purpose
- This is a high-throughput, low-latency distributed inference framework for generative AI and reasoning models. Its goal is to serve large language models (LLMs) using multiple backends (e.g., TRT-LLM, vLLM, SGLang) while efficiently managing GPU resources, routing requests, and handling KV cache offloading.
- The system is built in Rust for performance-critical components and in Python for extensibility. It supports multi-node environments, with key areas like LLM serving components and a distributed runtime.

##### File Structure to Identify Main Components
- Components Directory: Contains modules such as “components/http” (likely the OpenAI-compatible frontend server) and “components/metrics” (for instrumentation). These represent core service elements that directly handle request processing and system observation.
- Container & Deploy Folders: There are Dockerfiles, Kubernetes manifests (in deploy/Kubernetes and Helm charts) and docker-compose files. These files indicate containerization, orchestration, and the ability to deploy the platform in a distributed fashion.
- Deploy/dynamo: Encompasses the API server, operator, and database management. The operator specifically ties into Kubernetes CRDs and controllers, suggesting that the system uses an operator framework for lifecycle management.
- Launch Folder & Examples: Provide runnable samples and demo scripts (e.g., dynamo-run, llmctl) that illustrate how to start and interact with the system.

##### Main Components
a. Inference Serving & Frontends:
- OpenAI-Compatible HTTP API Server (components/http and lib/llm’s openai protocols)
- CLI Tools (from launch/ and examples/ directories) that allow local testing and serving.
b. Routing and Scheduling:
- Basic Router components and KV-aware routers (found in examples/llm/components and in lib/llm, particularly kv_router modules)
- Dynamic GPU and request scheduling logic
c. Worker / Backend Engines:
- Different backend engines (TRT-LLM, vLLM, SGLang, MistralRS) integrated in lib/llm/engines and container subdirectories.
- Worker modules responsible for inference computation.
d. Distributed Runtime and Orchestration:
- Distributed runtime services orchestrated via Docker Compose (deploy/docker-compose.yml) or Kubernetes deployments (deploy/Kubernetes, deploy/dynamo/operator).
- The Dynamo Operator handling CRDs and the deployment lifecycle.
- Additional deployment using Helm charts.
e. Monitoring and Metrics:
- Metrics components (in both components/metrics and deploy/metrics) that include Prometheus and Grafana dashboards.
f. SDK / Bindings:
- Language bindings (in lib/bindings) for Python, C, and C++ which facilitate integration with external systems.

##### Relationships and Data Flow
- How client requests (via REST API endpoints exposed by the frontend HTTP server) flow through the router.
- That requests are forwarded from the router to specialized backend workers (each one potentially running a different LLM engine).
- Connections to the KV caching layer; the design eliminates redundant KV cache recomputation.
- Communication with orchestration components (e.g., the operator managing deployments, the underlying message queues using protocols such as NATS, TCP, or even ZeroMQ as indicated by file names in lib/runtime).

##### Notable Architectural Patterns and Technologies
- Microservices/Modular design: Separating HTTP frontends, routers, and backend workers, each with clear responsibilities.
- Distributed system design: Focus on multi-node, containerized deployment using Docker, Kubernetes, and Helm.
- Polyglot approaches: Use of Rust for core performance, Python for extensibility, and C/C++ for language bindings.
- Dynamic request routing and KV cache management as key performance optimizations.
- Observability: Integrated metrics and logging as found in components/metrics and lib/runtime/logging, along with dashboards in deploy/metrics.

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client("Client"):::client

    subgraph "Frontend Layer"
        FE("HTTP Frontend"):::frontend
        subgraph "CLI Tools"
            CLI1("dynamo-run"):::cli
            CLI2("llmctl"):::cli
        end
    end

    subgraph "Routing & Scheduling"
        Router("Router / Scheduler"):::scheduler
    end

    subgraph "Inference Engine"
        BE("Backend Inference Engines"):::backend
        KV("KV Cache Manager"):::backend
    end

    subgraph "Distributed Runtime & Orchestration"
        OP("Operator"):::orchestration
        DC("Docker Compose Deployment"):::orchestration
        K8("Kubernetes Deployments"):::orchestration
    end

    subgraph "Monitoring & Metrics"
        Inst("Instrumentation"):::metrics
        Dash("Dashboards"):::metrics
    end

    subgraph "SDK / Bindings"
        SDK("SDK Bindings"):::sdk
    end

    Client -->|"REST"| FE
    FE -->|"routes"| Router
    Router -->|"forwards"| BE
    BE -->|"updates"| KV
    BE -->|"sendMetrics"| Inst
    Inst -->|"feedback"| Router
    CLI1 -->|"management"| OP
    CLI2 -->|"management"| OP
    OP -->|"manages"| FE
    OP -->|"controls"| BE
    OP -->|"deploys"| DC
    OP -->|"deploys"| K8
    SDK -->|"integration"| BE

    classDef frontend fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef scheduler fill:#D4E6F1,stroke:#2874A6,stroke-width:2px;
    classDef backend fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef orchestration fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef metrics fill:#F5CBA7,stroke:#CD853F,stroke-width:2px;
    classDef cli fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef sdk fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef client fill:#F5B7B1,stroke:#C0392B,stroke-width:2px;

    click FE "https://github.com/aryateja2106/nvidia-dynamo/tree/main/components/http"
    click CLI1 "https://github.com/aryateja2106/nvidia-dynamo/tree/main/launch/dynamo-run"
    click CLI2 "https://github.com/aryateja2106/nvidia-dynamo/tree/main/launch/llmctl"
    click Router "https://github.com/aryateja2106/nvidia-dynamo/tree/main/examples/llm/components"
    click Router "https://github.com/aryateja2106/nvidia-dynamo/tree/main/lib/llm/src/kv_router"
    click BE "https://github.com/aryateja2106/nvidia-dynamo/tree/main/lib/llm/src/engines"
    click DC "https://github.com/aryateja2106/nvidia-dynamo/blob/main/deploy/docker-compose.yml"
    click K8 "https://github.com/aryateja2106/nvidia-dynamo/tree/main/deploy/Kubernetes"
    click OP "https://github.com/aryateja2106/nvidia-dynamo/tree/main/deploy/dynamo/operator"
    click Inst "https://github.com/aryateja2106/nvidia-dynamo/tree/main/components/metrics"
    click Dash "https://github.com/aryateja2106/nvidia-dynamo/tree/main/deploy/metrics"
    click SDK "https://github.com/aryateja2106/nvidia-dynamo/tree/main/lib/bindings"
```
