How nvidia/k8s-device-plugin works
Main Components
• Kubernetes Cluster: Show the cluster environment where nodes run the DaemonSet.
• Device Plugin DaemonSet: Represent the NVIDIA device plugin deployed via DaemonSet. This includes different executable command binaries found in the “cmd” directory such as:
• nvidia-device-plugin (the main executable)
• gpu-feature-discovery (for automatic GPU labeling)
• mps-control-daemon (for MPS support)
• config-manager (to manage configurations)
• API and Configuration: Include the “api/config” packages that define the configuration version and flags that support both command-line and file-based configuration (supporting dynamic configuration via ConfigMap).
• Internal Libraries: Represent key internal packages that handle critical tasks:
• internal/cuda – dealing with CUDA operations and interfacing with drivers
• internal/mig – managing Multi-Instance GPU (MIG) functions
• internal/resource – managing GPU resource discovery and allocation details
• internal/plugin – orchestrating plugin-specific logic (including aspects of passing device specs, managing options, etc.)
• internal/rm – resource management with allocation, health checking, and configuration updates.
• Deployment Mechanisms: Show the integration with Helm for deployment and how the plugin leverages ConfigMaps for dynamic reconfiguration (which can be tailored per-node via node labels as described in the README).
• Underlying NVIDIA Components: Outside the containerized world, illustrate that the plugin interacts with the NVIDIA drivers, nvidia-container-runtime, and lower-level GPU device files (using concepts like “PASS_DEVICE_SPECS”, “DEVICE_LIST_STRATEGY”, etc.)
Relationships and Interactions
• Inside each node, show how the plugin (via its different executables) communicates with:
• The underlying container runtime (e.g., injecting environment variables or volume mounts to specify NVIDIA devices).
• The GPU drivers installed on the host (using configuration such as NVIDIA_DRIVER_ROOT).
• Use annotations or labels to indicate that various configuration options (MIG strategy, device ID strategy, etc.) come via command-line flags, environment variables, or external ConfigMaps (as shown via the “api/config”, “cmd/config-manager”, and “helm/deployments” directories).
• Integration with gpu-feature-discovery, which enriches nodes with GPU labels for scheduling.
• For the MPS and time-slicing mechanisms, highlight that there are distinct processes (e.g., mps-control-daemon and associated libraries) that modify how device allocations are handled.
Architectural Patterns or Design Principles
• Separation of concerns:
• The configuration layer (configuration files, flags, and ConfigMap logic).
• The resource discovery and management layer (inside internal/resource, internal/rm, internal/mig).
• The daemon/application layer (all binaries under the cmd directory).
• Present the deployment architecture (Helm chart driven, Kubernetes DaemonSet) demonstrating extensibility and multi-node support.
• Use of standard Kubernetes design (via device plugins) and integration with container runtimes (showing strategies like “envvar”, “volume-mounts”, “cdi-annotations”).
Relevant Technologies, Frameworks, and Libraries
• Golang for implementation (build scripts like Makefile and go.mod files show this).
• Docker (and container images) as evident from the Dockerfile in the “deployments/container” directory.
• Helm for deployment configuration.
• Kubernetes (device plugin API, DaemonSet, node labels, ConfigMap configuration).
• NVIDIA SDK components such as CUDA libraries and nvidia-container-runtime.