# nearai/private-ml-sdk architecture

> Architecture diagram of the GitHub repository nearai/private-ml-sdk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nearai/private-ml-sdk
- Repository: https://github.com/nearai/private-ml-sdk

## Overview

##### Project Type & Purpose
- This is a secure-computing SDK and toolchain for building and running Large Language Models inside Trusted Execution Environments (TEEs).
- It consists of two main subsystems:
–  The **image-build and runtime orchestration stack** (under `meta-dstack-nvidia`), which produces TDX/GPU-TEE VM images and runs them via dstack-vmm or the legacy `dstack.py`.
–  The **vllm-proxy service**, a standalone Python microservice for proxying LLM inference calls, with caching and signature verification.

##### High-Level Component Breakdown
1.  **Development/Build Tier**
- Host machine with Docker → runs `build.sh`, triggers Yocto/Bitbake in `meta-dstack-nvidia` layers → outputs VM images (`images/dstack-nvidia-*`).
- Local KMS provider (`meta-dstack-nvidia/dstack/key-provider-build/run.sh`) → exposes HTTPS endpoint for key derivation.
- e2e tests (`e2e/e2e.py`, `quote.py`) for validating quotes and workflows.
2.  **Orchestration & Runtime Tier**
- **dstack-vmm (or dstack.py)**
–  Consumes built VM images & KMS endpoint → launches Confidential VMs (CVMs) on TDX-enabled host + NVIDIA GPU TEE.
–  Performs remote attestation with client and KMS.
- **dstack-gateway (optional)**
–  Public-facing API and UI for managing CVMs.
–  Routes user traffic securely into the TEE-backed VMs.
3.  **Secure Compute & Application Tier**
- **CVM (Confidential VM)**
–  Intel TDX guest + NVIDIA GPU TEE
–  Runs Docker Compose (e.g. Jupyter service) inside secure enclave.
–  Mounts `/tapp/env-file` for secrets (via KMS).
- **Key Management Service (KMS)**
–  Derives, stores, and provides encryption keys.
–  Supports client-side env encryption (`--env-file`) and disk encryption.
- **vllm-proxy container** (optional side-service)
–  Exposes an HTTP API (`src/app/main.py`)
–  Implements caching layers (local in-memory, Redis) and signature verification.

##### Relationships & Data Flow
- **Orchestrator → CVM**: sequence: select image, assign GPU/CPU, launch.
- **Client (User/UI/CLI) → dstack-gateway/VMM CLI**: deployment commands and compose artifacts → CVM.
- **Client → CVM**: secure TLS traffic (Jupyter, custom apps).
- **vllm-proxy → Upstream LLM (OpenAI)**: API calls.
- **vllm-proxy → Redis**: cache read/write.

##### Architectural Patterns & Design Principles
- **Layered Build System**: Yocto layers (`meta-dstack-nvidia`, `meta-nvidia`, `meta-confidential-compute`, etc.) implementing separation of concerns (kernel vs. graphics vs. core recipes).
- **MicroVM Approach**: Container workloads run inside small TEE-backed VMs rather than full-blown host OS.
- **Zero-Trust & Attestation**: Remote attestation between client, KMS, and TEE guest ensures trust.
- **Separation of Concerns**: Orchestration logic (dstack) is separate from key management (KMS) and application runtime (jupyter, vllm-proxy).

##### Relevant Technologies & Frameworks
- Yocto/Bitbake (meta layers)
- Docker (build images, `docker-compose` inside TEE guest)
- NVIDIA GPU TEE (H100/H200/B100)
- Intel TDX
- dstack-vmm (Rust-based VMM orchestrator) or legacy Python `dstack.py`
- Python FastAPI/Flask (vllm-proxy)
- Redis (cache backend)
- AESMD (for local KMS provider)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Build/Dev Tier
    subgraph "Development/Build Tier"
        direction TB
        BuildScript["build.sh"]:::buildTool
        click BuildScript "https://github.com/nearai/private-ml-sdk/blob/main/./build.sh"

        YoctoLayers["Yocto/BitBake Layers"]:::buildTool
        click YoctoLayers "https://github.com/nearai/private-ml-sdk/tree/main/./meta-dstack-nvidia/"

        LocalConf["local.conf"]:::buildTool
        click LocalConf "https://github.com/nearai/private-ml-sdk/blob/main/./meta-dstack-nvidia/bb-build/conf/local.conf"

        ReproBuild["Reproducible Build Setup"]:::buildTool
        click ReproBuild "https://github.com/nearai/private-ml-sdk/tree/main/./meta-dstack-nvidia/repro-build/"

        SampleComposeBuild["docker-compose.yaml (CVM sample)"]:::buildTool
        click SampleComposeBuild "https://github.com/nearai/private-ml-sdk/blob/main/./meta-dstack-nvidia/scripts/samples/docker-compose.yaml"

        LegacyScript["dstack.py (Legacy Orchestrator)"]:::buildTool
        click LegacyScript "https://github.com/nearai/private-ml-sdk/blob/main/./meta-dstack-nvidia/scripts/bin/dstack.py"

        E2ETests["e2e.py / quote.py"]:::buildTool
        click E2ETests "https://github.com/nearai/private-ml-sdk/blob/main/./e2e/e2e.py"
        click E2ETests "https://github.com/nearai/private-ml-sdk/blob/main/./e2e/quote.py"
    end

    %% Orchestration & Runtime Tier
    subgraph "Orchestration & Runtime Tier"
        direction TB
        DstackVMM["dstack-vmm CLI (Rust)"]:::orchestrator
        click DstackVMM "https://github.com/nearai/private-ml-sdk/tree/main/./meta-dstack-nvidia/scripts/bin/dstack"

        HostAPI["Host-side Helper API"]:::orchestrator
        click HostAPI "https://github.com/nearai/private-ml-sdk/blob/main/./meta-dstack-nvidia/scripts/bin/host_api.py"

        SystemdUnit["cvm_dstack_sample.service"]:::orchestrator
        click SystemdUnit "https://github.com/nearai/private-ml-sdk/blob/main/./docs/cvm_dstack_sample.service"
    end

    %% Secure Compute & Application Tier
    subgraph "Secure Compute & Application Tier"
        direction TB
        CVM["CVM (TDX Guest + GPU TEE)"]:::secure
        KMS["KMS (HTTPS:3443)"]:::secure

        subgraph "Application Services"
            direction TB
            Jupyter["Jupyter Service"]:::secure
            VLLMProxy["vllm-proxy (FastAPI)"]:::app
            click VLLMProxy "https://github.com/nearai/private-ml-sdk/blob/main/./vllm-proxy/src/app/main.py"

            OpenAIProxy["OpenAI Proxy API"]:::app
            click OpenAIProxy "https://github.com/nearai/private-ml-sdk/blob/main/./vllm-proxy/src/app/api/v1/openai.py"

            RedisCache["Redis Cache"]:::app
            click RedisCache "https://github.com/nearai/private-ml-sdk/blob/main/./vllm-proxy/src/app/cache/redis.py"

            LocalCache["Local In-Memory Cache"]:::app
            click LocalCache "https://github.com/nearai/private-ml-sdk/blob/main/./vllm-proxy/src/app/cache/local_cache.py"

            QuoteVerify["Signature Verification"]:::app
            click QuoteVerify "https://github.com/nearai/private-ml-sdk/blob/main/./vllm-proxy/src/app/quote/quote.py"

            RedisCompose["redis/docker-compose.yaml"]:::app
            click RedisCompose "https://github.com/nearai/private-ml-sdk/blob/main/./vllm-proxy/redis/docker-compose.yaml"

            RedisRun["redis/run.sh"]:::app
            click RedisRun "https://github.com/nearai/private-ml-sdk/blob/main/./vllm-proxy/redis/run.sh"
        end
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        OpenAI["OpenAI (Upstream LLM)"]:::external
        DockerRegistry["Image Registry"]:::external
        Client["Client (UI/CLI)"]:::external
    end

    %% Connections - Build to Registry
    BuildScript -->|"build images"| DockerRegistry
    YoctoLayers --> DockerRegistry

    %% Orchestration flows
    Client -->|"deploy via CLI/UI"| DstackVMM
    Client -->|"deploy via UI/API"| SystemdUnit
    DstackVMM -->|"request keys & attestation"| KMS
    HostAPI -->|"assist launch"| DstackVMM
    DstackVMM -->|"launch VM"| CVM
    SystemdUnit -->|"systemd launch"| CVM
    KMS -->|"provide keys"| CVM

    %% Compute interactions
    CVM -->|"mount /tapp/env-file"| KMS
    CVM -->|"run compose"| Jupyter
    Jupyter -->|"user TLS traffic"| Client

    %% vllm-proxy interactions
    VLLMProxy -->|"cache R/W"| RedisCache
    VLLMProxy -->|"cache R/W"| LocalCache
    VLLMProxy -->|"verify signature"| QuoteVerify
    VLLMProxy -->|"proxy calls"| OpenAI
    VLLMProxy -->|"side-channel"| CVM

    %% Client direct
    Client -->|"TLS"| CVM

    %% Legend
    subgraph Legend
        direction LR
        B["Build/Dev Tools"]:::buildTool
        O["Orchestrator"]:::orchestrator
        S["Secure Compute"]:::secure
        A["App Services"]:::app
        E["External"]:::external
    end

    classDef buildTool fill:#bbe1fa,stroke:#1f618d,color:#1f618d;
    classDef orchestrator fill:#d5f5e3,stroke:#27ae60,color:#27ae60;
    classDef secure fill:#e8daef,stroke:#8e44ad,color:#8e44ad;
    classDef app fill:#fcf3cf,stroke:#e67e22,color:#e67e22;
    classDef external fill:#f0f3f4,stroke:#7f8c8d,color:#7f8c8d;
```
