How vrtust/k8s-vgpu-monitor works
Purpose and Runtime
k8s-vgpu-monitor is a node-local Linux C++11 Prometheus exporter for Kubernetes GPU-sharing environments. It observes NVIDIA GPU processes through NVML, maps PIDs to Kubernetes cgroups/Pods, aggregates usage per Pod and physical GPU, and exposes vGPU-oriented gauges on port 8080.
Primary executable and implementation: vgpu_monitor.cpp. The project is built as a native binary via CMakeLists.txt or build.sh.
Monitoring Agent Entry Point
The monitor is a long-running single process. Startup loads the GPU virtualization ratio, initializes NVML, logging, and prometheus-cpp exporter state, then enters a hardcoded periodic collection loop.
• vgpu_monitor.cpp
• gpu_allocation.txt
• CMakeLists.txt
The ratio file is a required runtime input and determines each Pod’s reported allocated vGPU memory as physical GPU memory divided by the ratio.
GPU Process Collection
Each polling cycle enumerates NVIDIA devices and active GPU processes using NVML. Per-process source measurements include GPU memory consumption and SM utilization, scoped to a physical GPU index.
• vgpu_monitor.cpp
This layer has a strict dependency on a host NVIDIA driver and NVML (libnvidia-ml.so), and must execute on nodes with visible NVIDIA GPUs.
Kubernetes Attribution Boundary
For every NVML PID, the agent reads Linux proc/cgroup state (/proc/<pid>/cgroup) and extracts Kubernetes pod or container identity from cgroup naming. This is the boundary between host-level GPU telemetry and Kubernetes workload attribution.
• vgpu_monitor.cpp
• kubernetes/vgpu_monitor.yml
Accurate attribution depends on host /proc visibility and permissions; the Kubernetes deployment is therefore architected as a per-node agent rather than a centralized service.
Pod-Level Aggregation and vGPU Conversion
Process measurements are grouped by Pod and GPU. The collector reports aggregated physical memory and SM use, plus calculated total vGPU memory allocation for each Pod/GPU pair. The intended Prometheus labels include gpu_id and pod.
• vgpu_monitor.cpp
• gpu_allocation.txt
The virtualization ratio is global configuration, not dynamically discovered from Kubernetes or a GPU virtualization provider.
Prometheus Export Boundary
Calculated values are written into prometheus-cpp Gauge families and served from the embedded exporter at :8080/metrics. Prometheus or another compatible scraper consumes this HTTP endpoint; the agent does not push metrics or require a control plane API.
• vgpu_monitor.cpp
• CMakeLists.txt
External runtime/build dependencies are NVML and preinstalled prometheus-cpp. CMake fetches simdjson and spdlog.
Container and Kubernetes Deployment
The repository supplies a container build and a Kubernetes DaemonSet manifest so one exporter runs on each eligible GPU node. The deployment must mount the allocation configuration and expose enough host access for /proc cgroup inspection and NVIDIA device/NVML access.
• dockerfile
• kubernetes/vgpu_monitor.yml
• gpu_allocation.txt
build.sh produces the local binary and writes an example ratio configuration into the build directory.
Components 14
Build & Packaging
- CMake build · Builds the C++11 monitor binary and resolves dependencies.
CMakeLists.txt - Build script · Builds locally and writes an example ratio configuration.
build.sh - Container image
dockerfile
Node Monitor
- GPU allocation ratio · Global ratio used to calculate allocated vGPU memory.
gpu_allocation.txt - vGPU monitor process · Long-running node-local polling agent.
vgpu_monitor.cpp - NVIDIA NVML · Requires visible NVIDIA GPUs, driver, and libnvidia-ml.so.
- GPU process telemetry · Enumerates devices and PID-scoped memory and SM usage.
vgpu_monitor.cpp - Host proc cgroups · Reads /proc/<pid>/cgroup to identify Kubernetes workloads.
- Pod/GPU aggregation · Groups process measurements by pod and physical GPU.
vgpu_monitor.cpp - vGPU gauge families · Exports physical usage and calculated allocation with pod and gpu_id labels.
vgpu_monitor.cpp - :8080/metrics
vgpu_monitor.cpp
Kubernetes Deployment
- GPU-node DaemonSet · Runs one agent per eligible node with host and NVIDIA access.
kubernetes/vgpu_monitor.yml - GPU node · Supplies GPU devices, NVML, and host proc visibility.
Other components
- Prometheus scraper · Pulls metrics; no push or control-plane API is used.
Connections 18
- CMake build → vGPU monitor process: builds
- Build script → CMake build: invokes
- Build script → GPU allocation ratio: writes example
- Container image → vGPU monitor process: packages
- GPU-node DaemonSet → Container image: runs image
- GPU-node DaemonSet → GPU allocation ratio: mounts
- GPU-node DaemonSet → GPU node: scheduled per node
- GPU node → vGPU monitor process: hosts
- GPU allocation ratio → vGPU monitor process: loads ratio
- vGPU monitor process → NVIDIA NVML: initializes and polls
- NVIDIA NVML → GPU process telemetry: device and PID data
- vGPU monitor process → Host proc cgroups: reads PID cgroups
- GPU process telemetry → Pod/GPU aggregation: process measurements
- Host proc cgroups → Pod/GPU aggregation: pod identity
- GPU allocation ratio → Pod/GPU aggregation: allocation conversion
- Pod/GPU aggregation → vGPU gauge families: aggregated values
- vGPU gauge families → :8080/metrics: serves gauges
- Prometheus scraper → :8080/metrics: scrapes