How nvidia/k8s-test-infra works
Mokka simulates NVIDIA GPU and surrounding infrastructure contracts on CPU-only Kubernetes nodes. A node agent reads configuration and stages host-visible driver, device, PCI, NVLink, fabric, and InfiniBand state; mock NVML/CUDA and related shims expose simulated behavior to software. Separately, the control plane reconciles declarative sGPU inventory and allocation into Kubernetes metadata and status, while the NRI plugin injects the mock overlay into workload containers. The README also documents fault injection, monitoring-consumer compatibility, topology, and IMEX readiness. Some subsystem wiring is unsampled, so uncertain links are omitted.
Main entry points: main.go Node agent · controller.go Reconciliation queues · agent.go Agent staging · manager.go Controller manager
Components 26
Runtime entrypoints
- Node agent
cmd/node-agent/main.go - NRI plugin
cmd/nri-plugin/main.go - Control plane
cmd/control-plane/main.go
Node simulation
- Agent staging
internal/agent/agent.go - Simulated state
internal/agent/state.go - CDI devices
internal/agent/cdi/spec.go - PCI sysfs
internal/pcisysfs/render.go - Kernel modules
internal/kmod/render.go
Device interfaces
- Mock NVML engine
pkg/gpu/mocknvml/engine/engine.go - NVML bridge
pkg/gpu/mocknvml/bridge/device.go - CUDA shim
shims/libcuda/bridge/cuda.go - Container injection
internal/nri/plugin.go - GPU injection rules
internal/nri/inject/gpu.go - Fault control
pkg/gpu/mockctl/publish.go
Kubernetes allocation
- Controller manager
internal/controlplane/manager.go - Reconciliation queues
internal/controlplane/controller/controller.go - Rack inventory
internal/sgpu/inventory/reconcile.go - Node projection
internal/sgpu/projection/controller.go - Aggregate status
internal/sgpu/status/controller.go
Fabric simulation
- InfiniBand daemon
internal/ib/daemon/server.go - InfiniBand sysfs
internal/ib/sysfs/render.go - Fabric Manager
internal/agent/fabricmanager/fabricmanager.go
Other components
- Cluster operator
- Workload container
- GPU software consumers
- Kubernetes API
Connections 28
- Cluster operator → Node agent: starts agent
- Node agent → Agent staging: runs
cmd/node-agent/main.go - Agent staging → Simulated state: stages state
- Node agent → PCI sysfs: configures
cmd/node-agent/main.go - Node agent → Kernel modules: configures
cmd/node-agent/main.go - Agent staging → CDI devices: stages devices
- GPU software consumers → NVML bridge: queries
- NVML bridge → Mock NVML engine: delegates
- Workload container → CUDA shim: loads shim
- CUDA shim → Mock NVML engine: uses simulation
- Node agent → NRI plugin: coordinates injection
- NRI plugin → Container injection: runs
cmd/nri-plugin/main.go - Container injection → GPU injection rules: adjusts containers
internal/nri/plugin.go - GPU injection rules → Workload container: injects overlay
- Cluster operator → Control plane: starts controller
- Control plane → Controller manager: runs
cmd/control-plane/main.go - Controller manager → Reconciliation queues: constructs
internal/controlplane/manager.go - Reconciliation queues → Rack inventory: reconciles
internal/controlplane/controller/controller.go - Reconciliation queues → Node projection: queues projection
internal/controlplane/controller/controller.go - Reconciliation queues → Aggregate status: queues status
internal/controlplane/controller/controller.go - Controller manager → Kubernetes API: creates clients
internal/controlplane/manager.go - Rack inventory → Kubernetes API: updates resources
- Node projection → Kubernetes API: patches Nodes
- Aggregate status → Kubernetes API: writes status
- Node agent → InfiniBand sysfs: configures
cmd/node-agent/main.go - InfiniBand sysfs → InfiniBand daemon: scans ports
internal/ib/daemon/server.go - Node agent → Fabric Manager: starts simulator
cmd/node-agent/main.go - Fault control → Mock NVML engine: sets mock state