How nvidia/gpu-operator works
Project Type and Purpose
• This is the NVIDIA GPU Operator, a Kubernetes Operator written in Go.
• Purpose: automate provisioning and lifecycle management of NVIDIA CUDA drivers, device plugins, container runtimes, DCGM-based monitoring, node labelling, and other GPU-related software components on a Kubernetes cluster via custom resources (CRDs).
File Structure
• api/ – Go types and code-generated clients for two CRDs: ClusterPolicy (v1) and NvidiaDriver (v1alpha1).
• controllers/ – Reconciliation loops (controllers) implementing the Operator business logic (clusterpolicy_controller, nvidiadriver_controller, upgrade_controller, state_manager, etc.).
• internal/ – Shared libraries: condition computation, rendering Kubernetes manifests, state management logic, validators, utilities.
• cmd/ – Three CLI binaries:
• gpu-operator (the main Operator binary)
• gpuop-cfg (validates and generates YAML for clusterpolicy & CSV)
• nvidia-validator (preflight/metrics validation)
• config/ and deployments/ / bundle/ / manifests/ – Kustomize, Helm charts, OLM bundle definitions, and static YAML assets used to install the Operator and its operands.
• assets/ – Per-component Kubernetes resources (ServiceAccounts, Roles, DaemonSets) for GPU driver installer, device plugin, DCGM exporter, MIG manager, vGPU manager, etc.
• tests/ – Unit tests, E2E framework, shell scripts, YAML workloads to validate behavior.
Extract Key Insights from README
• The Operator leverages the Kubernetes Device Plugin framework and Operator Framework.
• Administrators use a single standard OS image; the Operator deploys runtime components as containers.
• Scenarios: cloud or on-prem clusters that need rapid scaling of GPU nodes and automated software provisioning.
Components
a. Control Plane:
• Kubernetes API Server & etcd (cluster control plane).
• GPU Operator Pod (controller manager) with watchers for CRDs.
• CustomResourceDefinitions: ClusterPolicy and NvidiaDriver API groups.
b. Data Plane (Worker Nodes):
• DaemonSets for:
• NVIDIA Driver installer (privileged init).
• NVIDIA Container Runtime.
• Kubernetes Device Plugin for GPUs.
• DCGM Exporter (monitoring).
• Node Feature Discovery (optional chart).
• State managers (e.g., MIG, VFIO, Sandbox, MPS).
• ConfigMaps & Secrets (per-component configuration).
c. External Dependencies / Tools:
• Container registry (NVIDIA images).
• gpuop-cfg & nvidia-validator CLIs (admin-side).
• OLM / Helm (installer).
• Prometheus / Grafana (monitoring integration).
Relationships & Data Flows
• Operator Pod <–watches–> API Server for CR events (ClusterPolicy / NvidiaDriver).
• Operator Pod uses client-go to create/update child resources (DaemonSets, ConfigMaps, Roles).
• Worker nodes run DaemonSets which pull images from registry, install drivers, expose device plugins.
• DCGM exporter publishes GPU metrics to Prometheus.
• Admin CLIs validate and push CRs to the API Server.
Architectural Patterns & Design Principles
• Operator Pattern: declarative reconciliation loops.
• CRD-centric design: two custom API groups.
• Modular render library: templating YAML manifests.
• Kubernetes-native packaging: Helm charts, Kustomize, OLM bundle.
• Test automation: unit, integration, e2e frameworks.