# vrtust/k8s-vgpu-monitor architecture

> Architecture diagram of the GitHub repository vrtust/k8s-vgpu-monitor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vrtust/k8s-vgpu-monitor
- Repository: https://github.com/vrtust/k8s-vgpu-monitor

## Overview

#### Purpose and Runtime

`k8s-vgpu-monitor` is a node-local Linux C++11 Prometheus exporter for Kubernetes GPU-sharing environments. It observes NVIDIA GPU processes through NVML, maps PIDs to Kubernetes cgroups/Pods, aggregates usage per Pod and physical GPU, and exposes vGPU-oriented gauges on port 8080.

Primary executable and implementation: `vgpu_monitor.cpp`. The project is built as a native binary via `CMakeLists.txt` or `build.sh`.

#### Monitoring Agent Entry Point

The monitor is a long-running single process. Startup loads the GPU virtualization ratio, initializes NVML, logging, and prometheus-cpp exporter state, then enters a hardcoded periodic collection loop.

- `vgpu_monitor.cpp`
- `gpu_allocation.txt`
- `CMakeLists.txt`

The ratio file is a required runtime input and determines each Pod’s reported allocated vGPU memory as physical GPU memory divided by the ratio.

#### GPU Process Collection

Each polling cycle enumerates NVIDIA devices and active GPU processes using NVML. Per-process source measurements include GPU memory consumption and SM utilization, scoped to a physical GPU index.

- `vgpu_monitor.cpp`

This layer has a strict dependency on a host NVIDIA driver and NVML (`libnvidia-ml.so`), and must execute on nodes with visible NVIDIA GPUs.

#### Kubernetes Attribution Boundary

For every NVML PID, the agent reads Linux proc/cgroup state (`/proc/<pid>/cgroup`) and extracts Kubernetes pod or container identity from cgroup naming. This is the boundary between host-level GPU telemetry and Kubernetes workload attribution.

- `vgpu_monitor.cpp`
- `kubernetes/vgpu_monitor.yml`

Accurate attribution depends on host `/proc` visibility and permissions; the Kubernetes deployment is therefore architected as a per-node agent rather than a centralized service.

#### Pod-Level Aggregation and vGPU Conversion

Process measurements are grouped by Pod and GPU. The collector reports aggregated physical memory and SM use, plus calculated total vGPU memory allocation for each Pod/GPU pair. The intended Prometheus labels include `gpu_id` and `pod`.

- `vgpu_monitor.cpp`
- `gpu_allocation.txt`

The virtualization ratio is global configuration, not dynamically discovered from Kubernetes or a GPU virtualization provider.

#### Prometheus Export Boundary

Calculated values are written into prometheus-cpp Gauge families and served from the embedded exporter at `:8080/metrics`. Prometheus or another compatible scraper consumes this HTTP endpoint; the agent does not push metrics or require a control plane API.

- `vgpu_monitor.cpp`
- `CMakeLists.txt`

External runtime/build dependencies are NVML and preinstalled prometheus-cpp. CMake fetches simdjson and spdlog.

#### Container and Kubernetes Deployment

The repository supplies a container build and a Kubernetes DaemonSet manifest so one exporter runs on each eligible GPU node. The deployment must mount the allocation configuration and expose enough host access for `/proc` cgroup inspection and NVIDIA device/NVML access.

- `dockerfile`
- `kubernetes/vgpu_monitor.yml`
- `gpu_allocation.txt`

`build.sh` produces the local binary and writes an example ratio configuration into the build directory.

Main entry points: [`vgpu_monitor.cpp`](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp) (vGPU monitor process), [`gpu_allocation.txt`](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/gpu_allocation.txt) (GPU allocation ratio), [`kubernetes/vgpu_monitor.yml`](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/kubernetes/vgpu_monitor.yml) (GPU-node DaemonSet), [`build.sh`](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/build.sh) (Build script)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_build["Build &amp; Packaging"]
  node_cmake["CMake build<br/>native build<br/>[CMakeLists.txt]"]
  node_build_script["Build script<br/>build helper<br/>[build.sh]"]
  node_container_image["Container image<br/>container build"]
end

subgraph group_agent["Node Monitor"]
  node_ratio_config["GPU allocation ratio<br/>runtime configuration<br/>[gpu_allocation.txt]"]
  node_monitor{{"vGPU monitor process<br/>C++ exporter<br/>[vgpu_monitor.cpp]"}}
  node_nvml["NVIDIA NVML<br/>host GPU API"]
  node_gpu_processes["GPU process telemetry<br/>collection stage<br/>[vgpu_monitor.cpp]"]
  node_proc_cgroups["Host proc cgroups<br/>host attribution source"]
  node_pod_aggregation["Pod/GPU aggregation<br/>aggregation stage<br/>[vgpu_monitor.cpp]"]
  node_vgpu_metrics["vGPU gauge families<br/>Prometheus metrics<br/>[vgpu_monitor.cpp]"]
  node_metrics_endpoint[":8080/metrics<br/>embedded HTTP endpoint<br/>[vgpu_monitor.cpp]"]
end

subgraph group_deployment["Kubernetes Deployment"]
  node_daemonset["GPU-node DaemonSet<br/>Kubernetes workload<br/>[vgpu_monitor.yml]"]
  node_host_node["GPU node<br/>Kubernetes node"]
end

node_prometheus["Prometheus scraper<br/>external metrics consumer"]

node_cmake -->|"builds"| node_monitor
node_build_script -->|"invokes"| node_cmake
node_build_script -->|"writes example"| node_ratio_config
node_container_image -->|"packages"| node_monitor
node_daemonset -->|"runs image"| node_container_image
node_daemonset -->|"mounts"| node_ratio_config
node_daemonset -->|"scheduled per node"| node_host_node
node_host_node -->|"hosts"| node_monitor
node_ratio_config -->|"loads ratio"| node_monitor
node_monitor -->|"initializes and polls"| node_nvml
node_nvml -->|"device and PID data"| node_gpu_processes
node_monitor -->|"reads PID cgroups"| node_proc_cgroups
node_gpu_processes -->|"process measurements"| node_pod_aggregation
node_proc_cgroups -->|"pod identity"| node_pod_aggregation
node_ratio_config -->|"allocation conversion"| node_pod_aggregation
node_pod_aggregation -->|"aggregated values"| node_vgpu_metrics
node_vgpu_metrics -->|"serves gauges"| node_metrics_endpoint
node_prometheus -->|"scrapes"| node_metrics_endpoint

click node_cmake "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/CMakeLists.txt"
click node_build_script "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/build.sh"
click node_container_image "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/dockerfile"
click node_ratio_config "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/gpu_allocation.txt"
click node_monitor "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp"
click node_gpu_processes "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp"
click node_pod_aggregation "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp"
click node_vgpu_metrics "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp"
click node_metrics_endpoint "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp"
click node_daemonset "https://github.com/vrtust/k8s-vgpu-monitor/blob/main/kubernetes/vgpu_monitor.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cmake,node_build_script,node_container_image toneBlue
class node_ratio_config,node_monitor,node_nvml,node_gpu_processes,node_proc_cgroups,node_pod_aggregation,node_vgpu_metrics,node_metrics_endpoint toneAmber
class node_daemonset,node_host_node toneMint
class node_prometheus toneNeutral
```

## Components

### Build & Packaging

- [CMake build](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/CMakeLists.txt): native build · Builds the C++11 monitor binary and resolves dependencies. · `CMakeLists.txt`
- [Build script](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/build.sh): build helper · Builds locally and writes an example ratio configuration. · `build.sh`
- [Container image](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/dockerfile): container build · `dockerfile`

### Node Monitor

- [GPU allocation ratio](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/gpu_allocation.txt): runtime configuration · Global ratio used to calculate allocated vGPU memory. · `gpu_allocation.txt`
- [vGPU monitor process](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp): C++ exporter · Long-running node-local polling agent. · `vgpu_monitor.cpp`
- **NVIDIA NVML**: host GPU API · Requires visible NVIDIA GPUs, driver, and libnvidia-ml.so.
- [GPU process telemetry](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp): collection stage · Enumerates devices and PID-scoped memory and SM usage. · `vgpu_monitor.cpp`
- **Host proc cgroups**: host attribution source · Reads /proc/<pid>/cgroup to identify Kubernetes workloads.
- [Pod/GPU aggregation](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp): aggregation stage · Groups process measurements by pod and physical GPU. · `vgpu_monitor.cpp`
- [vGPU gauge families](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp): Prometheus metrics · Exports physical usage and calculated allocation with pod and gpu_id labels. · `vgpu_monitor.cpp`
- [:8080/metrics](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/vgpu_monitor.cpp): embedded HTTP endpoint · `vgpu_monitor.cpp`

### Kubernetes Deployment

- [GPU-node DaemonSet](https://github.com/vrtust/k8s-vgpu-monitor/blob/main/kubernetes/vgpu_monitor.yml): Kubernetes workload · Runs one agent per eligible node with host and NVIDIA access. · `kubernetes/vgpu_monitor.yml`
- **GPU node**: Kubernetes node · Supplies GPU devices, NVML, and host proc visibility.

### Other components

- **Prometheus scraper**: external metrics consumer · Pulls metrics; no push or control-plane API is used.

## Connections

- CMake build → vGPU monitor process: builds
- Build script → CMake build: invokes
- Build script → GPU allocation ratio: writes example
- Container image → vGPU monitor process: packages
- GPU-node DaemonSet → Container image: runs image
- GPU-node DaemonSet → GPU allocation ratio: mounts
- GPU-node DaemonSet → GPU node: scheduled per node
- GPU node → vGPU monitor process: hosts
- GPU allocation ratio → vGPU monitor process: loads ratio
- vGPU monitor process → NVIDIA NVML: initializes and polls
- NVIDIA NVML → GPU process telemetry: device and PID data
- vGPU monitor process → Host proc cgroups: reads PID cgroups
- GPU process telemetry → Pod/GPU aggregation: process measurements
- Host proc cgroups → Pod/GPU aggregation: pod identity
- GPU allocation ratio → Pod/GPU aggregation: allocation conversion
- Pod/GPU aggregation → vGPU gauge families: aggregated values
- vGPU gauge families → :8080/metrics: serves gauges
- Prometheus scraper → :8080/metrics: scrapes
