# nvidia/k8s-test-infra architecture

> Architecture diagram of the GitHub repository nvidia/k8s-test-infra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/k8s-test-infra
- Repository: https://github.com/nvidia/k8s-test-infra

## Overview

Mokka simulates NVIDIA GPU and surrounding infrastructure contracts on CPU-only Kubernetes nodes. A node agent reads configuration and stages host-visible driver, device, PCI, NVLink, fabric, and InfiniBand state; mock NVML/CUDA and related shims expose simulated behavior to software. Separately, the control plane reconciles declarative sGPU inventory and allocation into Kubernetes metadata and status, while the NRI plugin injects the mock overlay into workload containers. The README also documents fault injection, monitoring-consumer compatibility, topology, and IMEX readiness. Some subsystem wiring is unsampled, so uncertain links are omitted.

Main entry points: [`cmd/node-agent/main.go`](https://github.com/nvidia/k8s-test-infra/blob/main/cmd/node-agent/main.go) (Node agent), [`internal/controlplane/controller/controller.go`](https://github.com/nvidia/k8s-test-infra/blob/main/internal/controlplane/controller/controller.go) (Reconciliation queues), [`internal/agent/agent.go`](https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/agent.go) (Agent staging), [`internal/controlplane/manager.go`](https://github.com/nvidia/k8s-test-infra/blob/main/internal/controlplane/manager.go) (Controller manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Runtime entrypoints"]
  node_node_cli["Node agent<br/>[main.go]"]
  node_nri_cli["NRI plugin<br/>[main.go]"]
  node_cp_cli["Control plane<br/>[main.go]"]
end

subgraph group_node["Node simulation"]
  node_agent_core["Agent staging<br/>[agent.go]"]
  node_agent_state["Simulated state<br/>[state.go]"]
  node_cdi["CDI devices<br/>[spec.go]"]
  node_pci["PCI sysfs<br/>[render.go]"]
  node_kernel["Kernel modules<br/>[render.go]"]
end

subgraph group_mock["Device interfaces"]
  node_nvml_engine["Mock NVML engine<br/>[engine.go]"]
  node_nvml_bridge["NVML bridge<br/>[device.go]"]
  node_cuda["CUDA shim<br/>[cuda.go]"]
  node_nri_plugin["Container injection<br/>[plugin.go]"]
  node_nri_gpu["GPU injection rules<br/>[gpu.go]"]
  node_mockctl["Fault control<br/>[publish.go]"]
end

subgraph group_control["Kubernetes allocation"]
  node_cp_manager["Controller manager<br/>[manager.go]"]
  node_cp_controller["Reconciliation queues<br/>[controller.go]"]
  node_inventory["Rack inventory<br/>[reconcile.go]"]
  node_projection["Node projection<br/>[controller.go]"]
  node_status["Aggregate status<br/>[controller.go]"]
end

subgraph group_fabric["Fabric simulation"]
  node_ib_daemon["InfiniBand daemon<br/>[server.go]"]
  node_ib_sysfs["InfiniBand sysfs<br/>[render.go]"]
  node_fabric_manager["Fabric Manager<br/>[fabricmanager.go]"]
end

node_operator(("Cluster operator"))
node_workload(("Workload container"))
node_gpu_consumers(("GPU software consumers"))
node_kube[("Kubernetes API")]

node_operator -.->|"starts agent"| node_node_cli
node_node_cli -->|"runs"| node_agent_core
node_agent_core -.->|"stages state"| node_agent_state
node_node_cli -->|"configures"| node_pci
node_node_cli -->|"configures"| node_kernel
node_agent_core -.->|"stages devices"| node_cdi
node_gpu_consumers -.->|"queries"| node_nvml_bridge
node_nvml_bridge -.->|"delegates"| node_nvml_engine
node_workload -.->|"loads shim"| node_cuda
node_cuda -.->|"uses simulation"| node_nvml_engine
node_node_cli -.->|"coordinates injection"| node_nri_cli
node_nri_cli -->|"runs"| node_nri_plugin
node_nri_plugin -->|"adjusts containers"| node_nri_gpu
node_nri_gpu -.->|"injects overlay"| node_workload
node_operator -.->|"starts controller"| node_cp_cli
node_cp_cli -->|"runs"| node_cp_manager
node_cp_manager -->|"constructs"| node_cp_controller
node_cp_controller -->|"reconciles"| node_inventory
node_cp_controller -->|"queues projection"| node_projection
node_cp_controller -->|"queues status"| node_status
node_cp_manager -->|"creates clients"| node_kube
node_inventory -.->|"updates resources"| node_kube
node_projection -.->|"patches Nodes"| node_kube
node_status -.->|"writes status"| node_kube
node_node_cli -->|"configures"| node_ib_sysfs
node_ib_sysfs -->|"scans ports"| node_ib_daemon
node_node_cli -->|"starts simulator"| node_fabric_manager
node_mockctl -.->|"sets mock state"| node_nvml_engine

click node_node_cli "https://github.com/nvidia/k8s-test-infra/blob/main/cmd/node-agent/main.go"
click node_agent_core "https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/agent.go"
click node_agent_state "https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/state.go"
click node_cdi "https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/cdi/spec.go"
click node_pci "https://github.com/nvidia/k8s-test-infra/blob/main/internal/pcisysfs/render.go"
click node_kernel "https://github.com/nvidia/k8s-test-infra/blob/main/internal/kmod/render.go"
click node_nvml_engine "https://github.com/nvidia/k8s-test-infra/blob/main/pkg/gpu/mocknvml/engine/engine.go"
click node_nvml_bridge "https://github.com/nvidia/k8s-test-infra/blob/main/pkg/gpu/mocknvml/bridge/device.go"
click node_cuda "https://github.com/nvidia/k8s-test-infra/blob/main/shims/libcuda/bridge/cuda.go"
click node_nri_cli "https://github.com/nvidia/k8s-test-infra/blob/main/cmd/nri-plugin/main.go"
click node_nri_plugin "https://github.com/nvidia/k8s-test-infra/blob/main/internal/nri/plugin.go"
click node_nri_gpu "https://github.com/nvidia/k8s-test-infra/blob/main/internal/nri/inject/gpu.go"
click node_cp_cli "https://github.com/nvidia/k8s-test-infra/blob/main/cmd/control-plane/main.go"
click node_cp_manager "https://github.com/nvidia/k8s-test-infra/blob/main/internal/controlplane/manager.go"
click node_cp_controller "https://github.com/nvidia/k8s-test-infra/blob/main/internal/controlplane/controller/controller.go"
click node_inventory "https://github.com/nvidia/k8s-test-infra/blob/main/internal/sgpu/inventory/reconcile.go"
click node_projection "https://github.com/nvidia/k8s-test-infra/blob/main/internal/sgpu/projection/controller.go"
click node_status "https://github.com/nvidia/k8s-test-infra/blob/main/internal/sgpu/status/controller.go"
click node_ib_daemon "https://github.com/nvidia/k8s-test-infra/blob/main/internal/ib/daemon/server.go"
click node_ib_sysfs "https://github.com/nvidia/k8s-test-infra/blob/main/internal/ib/sysfs/render.go"
click node_fabric_manager "https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/fabricmanager/fabricmanager.go"
click node_mockctl "https://github.com/nvidia/k8s-test-infra/blob/main/pkg/gpu/mockctl/publish.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_cli,node_nri_cli,node_cp_cli toneBlue
class node_agent_core,node_agent_state,node_cdi,node_pci,node_kernel,node_kube toneAmber
class node_nvml_engine,node_nvml_bridge,node_cuda,node_nri_plugin,node_nri_gpu,node_mockctl toneMint
class node_cp_manager,node_cp_controller,node_inventory,node_projection,node_status toneRose
class node_ib_daemon,node_ib_sysfs,node_fabric_manager,node_operator,node_workload,node_gpu_consumers toneIndigo
```

## Components

### Runtime entrypoints

- [Node agent](https://github.com/nvidia/k8s-test-infra/blob/main/cmd/node-agent/main.go): `cmd/node-agent/main.go`
- [NRI plugin](https://github.com/nvidia/k8s-test-infra/blob/main/cmd/nri-plugin/main.go): `cmd/nri-plugin/main.go`
- [Control plane](https://github.com/nvidia/k8s-test-infra/blob/main/cmd/control-plane/main.go): `cmd/control-plane/main.go`

### Node simulation

- [Agent staging](https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/agent.go): `internal/agent/agent.go`
- [Simulated state](https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/state.go): `internal/agent/state.go`
- [CDI devices](https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/cdi/spec.go): `internal/agent/cdi/spec.go`
- [PCI sysfs](https://github.com/nvidia/k8s-test-infra/blob/main/internal/pcisysfs/render.go): `internal/pcisysfs/render.go`
- [Kernel modules](https://github.com/nvidia/k8s-test-infra/blob/main/internal/kmod/render.go): `internal/kmod/render.go`

### Device interfaces

- [Mock NVML engine](https://github.com/nvidia/k8s-test-infra/blob/main/pkg/gpu/mocknvml/engine/engine.go): `pkg/gpu/mocknvml/engine/engine.go`
- [NVML bridge](https://github.com/nvidia/k8s-test-infra/blob/main/pkg/gpu/mocknvml/bridge/device.go): `pkg/gpu/mocknvml/bridge/device.go`
- [CUDA shim](https://github.com/nvidia/k8s-test-infra/blob/main/shims/libcuda/bridge/cuda.go): `shims/libcuda/bridge/cuda.go`
- [Container injection](https://github.com/nvidia/k8s-test-infra/blob/main/internal/nri/plugin.go): `internal/nri/plugin.go`
- [GPU injection rules](https://github.com/nvidia/k8s-test-infra/blob/main/internal/nri/inject/gpu.go): `internal/nri/inject/gpu.go`
- [Fault control](https://github.com/nvidia/k8s-test-infra/blob/main/pkg/gpu/mockctl/publish.go): `pkg/gpu/mockctl/publish.go`

### Kubernetes allocation

- [Controller manager](https://github.com/nvidia/k8s-test-infra/blob/main/internal/controlplane/manager.go): `internal/controlplane/manager.go`
- [Reconciliation queues](https://github.com/nvidia/k8s-test-infra/blob/main/internal/controlplane/controller/controller.go): `internal/controlplane/controller/controller.go`
- [Rack inventory](https://github.com/nvidia/k8s-test-infra/blob/main/internal/sgpu/inventory/reconcile.go): `internal/sgpu/inventory/reconcile.go`
- [Node projection](https://github.com/nvidia/k8s-test-infra/blob/main/internal/sgpu/projection/controller.go): `internal/sgpu/projection/controller.go`
- [Aggregate status](https://github.com/nvidia/k8s-test-infra/blob/main/internal/sgpu/status/controller.go): `internal/sgpu/status/controller.go`

### Fabric simulation

- [InfiniBand daemon](https://github.com/nvidia/k8s-test-infra/blob/main/internal/ib/daemon/server.go): `internal/ib/daemon/server.go`
- [InfiniBand sysfs](https://github.com/nvidia/k8s-test-infra/blob/main/internal/ib/sysfs/render.go): `internal/ib/sysfs/render.go`
- [Fabric Manager](https://github.com/nvidia/k8s-test-infra/blob/main/internal/agent/fabricmanager/fabricmanager.go): `internal/agent/fabricmanager/fabricmanager.go`

### Other components

- **Cluster operator**
- **Workload container**
- **GPU software consumers**
- **Kubernetes API**

## Connections

- Cluster operator → Node agent: starts agent
- Node agent → Agent staging: runs (evidence: `cmd/node-agent/main.go`)
- Agent staging → Simulated state: stages state
- Node agent → PCI sysfs: configures (evidence: `cmd/node-agent/main.go`)
- Node agent → Kernel modules: configures (evidence: `cmd/node-agent/main.go`)
- Agent staging → CDI devices: stages devices
- GPU software consumers → NVML bridge: queries
- NVML bridge → Mock NVML engine: delegates
- Workload container → CUDA shim: loads shim
- CUDA shim → Mock NVML engine: uses simulation
- Node agent → NRI plugin: coordinates injection
- NRI plugin → Container injection: runs (evidence: `cmd/nri-plugin/main.go`)
- Container injection → GPU injection rules: adjusts containers (evidence: `internal/nri/plugin.go`)
- GPU injection rules → Workload container: injects overlay
- Cluster operator → Control plane: starts controller
- Control plane → Controller manager: runs (evidence: `cmd/control-plane/main.go`)
- Controller manager → Reconciliation queues: constructs (evidence: `internal/controlplane/manager.go`)
- Reconciliation queues → Rack inventory: reconciles (evidence: `internal/controlplane/controller/controller.go`)
- Reconciliation queues → Node projection: queues projection (evidence: `internal/controlplane/controller/controller.go`)
- Reconciliation queues → Aggregate status: queues status (evidence: `internal/controlplane/controller/controller.go`)
- Controller manager → Kubernetes API: creates clients (evidence: `internal/controlplane/manager.go`)
- Rack inventory → Kubernetes API: updates resources
- Node projection → Kubernetes API: patches Nodes
- Aggregate status → Kubernetes API: writes status
- Node agent → InfiniBand sysfs: configures (evidence: `cmd/node-agent/main.go`)
- InfiniBand sysfs → InfiniBand daemon: scans ports (evidence: `internal/ib/daemon/server.go`)
- Node agent → Fabric Manager: starts simulator (evidence: `cmd/node-agent/main.go`)
- Fault control → Mock NVML engine: sets mock state
