How nearai/private-ml-sdk works
Project Type & Purpose
• This is a secure-computing SDK and toolchain for building and running Large Language Models inside Trusted Execution Environments (TEEs).
• It consists of two main subsystems:
– The image-build and runtime orchestration stack (under meta-dstack-nvidia), which produces TDX/GPU-TEE VM images and runs them via dstack-vmm or the legacy dstack.py.
– The vllm-proxy service, a standalone Python microservice for proxying LLM inference calls, with caching and signature verification.
High-Level Component Breakdown
1. Development/Build Tier
• Host machine with Docker → runs build.sh, triggers Yocto/Bitbake in meta-dstack-nvidia layers → outputs VM images (images/dstack-nvidia-*).
• Local KMS provider (meta-dstack-nvidia/dstack/key-provider-build/run.sh) → exposes HTTPS endpoint for key derivation.
• e2e tests (e2e/e2e.py, quote.py) for validating quotes and workflows.
2. Orchestration & Runtime Tier
• dstack-vmm (or dstack.py)
– Consumes built VM images & KMS endpoint → launches Confidential VMs (CVMs) on TDX-enabled host + NVIDIA GPU TEE.
– Performs remote attestation with client and KMS.
• dstack-gateway (optional)
– Public-facing API and UI for managing CVMs.
– Routes user traffic securely into the TEE-backed VMs.
3. Secure Compute & Application Tier
• CVM (Confidential VM)
– Intel TDX guest + NVIDIA GPU TEE
– Runs Docker Compose (e.g. Jupyter service) inside secure enclave.
– Mounts /tapp/env-file for secrets (via KMS).
• Key Management Service (KMS)
– Derives, stores, and provides encryption keys.
– Supports client-side env encryption (--env-file) and disk encryption.
• vllm-proxy container (optional side-service)
– Exposes an HTTP API (src/app/main.py)
– Implements caching layers (local in-memory, Redis) and signature verification.
Relationships & Data Flow
• Orchestrator → CVM: sequence: select image, assign GPU/CPU, launch.
• Client (User/UI/CLI) → dstack-gateway/VMM CLI: deployment commands and compose artifacts → CVM.
• Client → CVM: secure TLS traffic (Jupyter, custom apps).
• vllm-proxy → Upstream LLM (OpenAI): API calls.
• vllm-proxy → Redis: cache read/write.
Architectural Patterns & Design Principles
• Layered Build System: Yocto layers (meta-dstack-nvidia, meta-nvidia, meta-confidential-compute, etc.) implementing separation of concerns (kernel vs. graphics vs. core recipes).
• MicroVM Approach: Container workloads run inside small TEE-backed VMs rather than full-blown host OS.
• Zero-Trust & Attestation: Remote attestation between client, KMS, and TEE guest ensures trust.
• Separation of Concerns: Orchestration logic (dstack) is separate from key management (KMS) and application runtime (jupyter, vllm-proxy).
Relevant Technologies & Frameworks
• Yocto/Bitbake (meta layers)
• Docker (build images, docker-compose inside TEE guest)
• NVIDIA GPU TEE (H100/H200/B100)
• Intel TDX
• dstack-vmm (Rust-based VMM orchestrator) or legacy Python dstack.py
• Python FastAPI/Flask (vllm-proxy)
• Redis (cache backend)
• AESMD (for local KMS provider)