# nvidia/nvsentinel architecture

> Architecture diagram of the GitHub repository nvidia/nvsentinel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/nvsentinel
- Repository: https://github.com/nvidia/nvsentinel

## Overview

NVSentinel is a modular Kubernetes GPU-fault detection and repair platform. Its documented user-to-result path is monitoring faults, isolating affected nodes through cordon and optional drain, then optionally repairing them with a GPU reset or reboot; preflight adds an active readiness check before workloads start. The README describes modules coordinating through a shared MongoDB event store and Kubernetes API, not direct module-to-module calls. The architecture excerpt is truncated and most module internals were not sampled, so uncertain links are dashed and uncited rather than presented as inspected call paths.

Main entry points: [`store-client/pkg/client/postgresql_pipeline_builder.go`](https://github.com/nvidia/nvsentinel/blob/main/store-client/pkg/client/postgresql_pipeline_builder.go) (Event Store), [`platform-connectors/main.go`](https://github.com/nvidia/nvsentinel/blob/main/platform-connectors/main.go) (Platform Connectors), [`janitor/main.go`](https://github.com/nvidia/nvsentinel/blob/main/janitor/main.go) (Janitor Controllers), [`janitor-provider/main.go`](https://github.com/nvidia/nvsentinel/blob/main/janitor-provider/main.go) (CSP Provider)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_observe["Fault Detection"]
  node_gpu_monitor["GPU Monitor<br/>[dcgm.py]"]
  node_syslog_monitor["Syslog Monitor<br/>[main.go]"]
  node_nic_monitor["NIC Monitor<br/>[main.go]"]
  node_csp_monitor["CSP Monitor<br/>[main.go]"]
  node_object_monitor["Object Monitor<br/>[main.go]"]
  node_health_analyzer["Event Analyzer<br/>[main.go]"]
end

subgraph group_ingest["Event Ingestion"]
  node_platform_connectors["Platform Connectors<br/>[main.go]"]
  node_event_exporter["Event Exporter<br/>[main.go]"]
  node_event_store[("Event Store")]
end

subgraph group_protect["Node Protection"]
  node_quarantine["Fault Quarantine<br/>[main.go]"]
  node_node_drainer["Node Drainer<br/>[main.go]"]
end

subgraph group_repair["Remediation"]
  node_fault_remediation["Fault Remediation<br/>[main.go]"]
  node_janitor["Janitor Controllers<br/>[main.go]"]
  node_gpu_reset["GPU Reset"]
  node_janitor_provider["CSP Provider<br/>[main.go]"]
  node_lifecycle_manager["Lifecycle Manager<br/>[main.go]"]
end

subgraph group_workload["Workload Readiness"]
  node_preflight["Preflight<br/>[main.go]"]
  node_gang_controller["Gang Checks<br/>[gang_controller.go]"]
end

node_workload(("Kubernetes Workload"))
node_kubernetes[("Kubernetes API")]
node_cloud_provider["Cloud Provider"]

node_gpu_monitor -.->|"publishes events"| node_platform_connectors
node_syslog_monitor -.->|"publishes events"| node_platform_connectors
node_nic_monitor -.->|"publishes events"| node_platform_connectors
node_csp_monitor -.->|"reports maintenance"| node_event_store
node_object_monitor -.->|"reports faults"| node_event_store
node_platform_connectors -.->|"stores events"| node_event_store
node_event_store -.->|"supplies events"| node_health_analyzer
node_event_store -.->|"exports events"| node_event_exporter
node_event_store -.->|"provides events"| node_quarantine
node_event_store -.->|"provides events"| node_fault_remediation
node_quarantine -.->|"cordons node"| node_kubernetes
node_node_drainer -.->|"drains workloads"| node_kubernetes
node_fault_remediation -.->|"requests repair"| node_janitor
node_janitor -->|"reconciles reset"| node_gpu_reset
node_janitor -.->|"requests reboot"| node_janitor_provider
node_janitor_provider -->|"signals repair"| node_cloud_provider
node_janitor_provider -->|"reads node"| node_kubernetes
node_lifecycle_manager -.->|"validates nodes"| node_kubernetes
node_workload -.->|"runs readiness check"| node_preflight
node_preflight -->|"coordinates checks"| node_gang_controller
node_gang_controller -->|"reads pod state"| node_kubernetes

click node_gpu_monitor "https://github.com/nvidia/nvsentinel/blob/main/health-monitors/gpu-health-monitor/gpu_health_monitor/dcgm_watcher/dcgm.py"
click node_syslog_monitor "https://github.com/nvidia/nvsentinel/blob/main/health-monitors/syslog-health-monitor/main.go"
click node_nic_monitor "https://github.com/nvidia/nvsentinel/blob/main/health-monitors/nic-health-monitor/main.go"
click node_csp_monitor "https://github.com/nvidia/nvsentinel/blob/main/health-monitors/csp-health-monitor/cmd/csp-health-monitor/main.go"
click node_object_monitor "https://github.com/nvidia/nvsentinel/blob/main/health-monitors/kubernetes-object-monitor/main.go"
click node_health_analyzer "https://github.com/nvidia/nvsentinel/blob/main/health-events-analyzer/main.go"
click node_platform_connectors "https://github.com/nvidia/nvsentinel/blob/main/platform-connectors/main.go"
click node_event_exporter "https://github.com/nvidia/nvsentinel/blob/main/event-exporter/main.go"
click node_event_store "https://github.com/nvidia/nvsentinel/blob/main/store-client/pkg/client/postgresql_pipeline_builder.go"
click node_quarantine "https://github.com/nvidia/nvsentinel/blob/main/fault-quarantine/main.go"
click node_node_drainer "https://github.com/nvidia/nvsentinel/blob/main/node-drainer/main.go"
click node_fault_remediation "https://github.com/nvidia/nvsentinel/blob/main/fault-remediation/main.go"
click node_janitor "https://github.com/nvidia/nvsentinel/blob/main/janitor/main.go"
click node_gpu_reset "https://github.com/nvidia/nvsentinel/blob/main/janitor/pkg/controller/gpureset_controller.go"
click node_janitor_provider "https://github.com/nvidia/nvsentinel/blob/main/janitor-provider/main.go"
click node_lifecycle_manager "https://github.com/nvidia/nvsentinel/blob/main/lifecycle-manager/main.go"
click node_preflight "https://github.com/nvidia/nvsentinel/blob/main/preflight/main.go"
click node_gang_controller "https://github.com/nvidia/nvsentinel/blob/main/preflight/pkg/controller/gang_controller.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gpu_monitor,node_syslog_monitor,node_nic_monitor,node_csp_monitor,node_object_monitor,node_health_analyzer toneBlue
class node_platform_connectors,node_event_exporter,node_event_store,node_kubernetes toneAmber
class node_quarantine,node_node_drainer toneMint
class node_fault_remediation,node_janitor,node_gpu_reset,node_janitor_provider,node_lifecycle_manager toneRose
class node_preflight,node_gang_controller,node_workload,node_cloud_provider toneIndigo
```

## Components

### Fault Detection

- [GPU Monitor](https://github.com/nvidia/nvsentinel/blob/main/health-monitors/gpu-health-monitor/gpu_health_monitor/dcgm_watcher/dcgm.py): `health-monitors/gpu-health-monitor/gpu_health_monitor/dcgm_watcher/dcgm.py`
- [Syslog Monitor](https://github.com/nvidia/nvsentinel/blob/main/health-monitors/syslog-health-monitor/main.go): `health-monitors/syslog-health-monitor/main.go`
- [NIC Monitor](https://github.com/nvidia/nvsentinel/blob/main/health-monitors/nic-health-monitor/main.go): `health-monitors/nic-health-monitor/main.go`
- [CSP Monitor](https://github.com/nvidia/nvsentinel/blob/main/health-monitors/csp-health-monitor/cmd/csp-health-monitor/main.go): `health-monitors/csp-health-monitor/cmd/csp-health-monitor/main.go`
- [Object Monitor](https://github.com/nvidia/nvsentinel/blob/main/health-monitors/kubernetes-object-monitor/main.go): `health-monitors/kubernetes-object-monitor/main.go`
- [Event Analyzer](https://github.com/nvidia/nvsentinel/blob/main/health-events-analyzer/main.go): `health-events-analyzer/main.go`

### Event Ingestion

- [Platform Connectors](https://github.com/nvidia/nvsentinel/blob/main/platform-connectors/main.go): `platform-connectors/main.go`
- [Event Exporter](https://github.com/nvidia/nvsentinel/blob/main/event-exporter/main.go): `event-exporter/main.go`
- [Event Store](https://github.com/nvidia/nvsentinel/blob/main/store-client/pkg/client/postgresql_pipeline_builder.go): `store-client/pkg/client/postgresql_pipeline_builder.go`

### Node Protection

- [Fault Quarantine](https://github.com/nvidia/nvsentinel/blob/main/fault-quarantine/main.go): `fault-quarantine/main.go`
- [Node Drainer](https://github.com/nvidia/nvsentinel/blob/main/node-drainer/main.go): `node-drainer/main.go`

### Remediation

- [Fault Remediation](https://github.com/nvidia/nvsentinel/blob/main/fault-remediation/main.go): `fault-remediation/main.go`
- [Janitor Controllers](https://github.com/nvidia/nvsentinel/blob/main/janitor/main.go): `janitor/main.go`
- [GPU Reset](https://github.com/nvidia/nvsentinel/blob/main/janitor/pkg/controller/gpureset_controller.go): `janitor/pkg/controller/gpureset_controller.go`
- [CSP Provider](https://github.com/nvidia/nvsentinel/blob/main/janitor-provider/main.go): `janitor-provider/main.go`
- [Lifecycle Manager](https://github.com/nvidia/nvsentinel/blob/main/lifecycle-manager/main.go): `lifecycle-manager/main.go`

### Workload Readiness

- [Preflight](https://github.com/nvidia/nvsentinel/blob/main/preflight/main.go): `preflight/main.go`
- [Gang Checks](https://github.com/nvidia/nvsentinel/blob/main/preflight/pkg/controller/gang_controller.go): `preflight/pkg/controller/gang_controller.go`

### Other components

- **Kubernetes Workload**
- **Kubernetes API**
- **Cloud Provider**

## Connections

- GPU Monitor → Platform Connectors: publishes events
- Syslog Monitor → Platform Connectors: publishes events
- NIC Monitor → Platform Connectors: publishes events
- CSP Monitor → Event Store: reports maintenance
- Object Monitor → Event Store: reports faults
- Platform Connectors → Event Store: stores events
- Event Store → Event Analyzer: supplies events
- Event Store → Event Exporter: exports events
- Event Store → Fault Quarantine: provides events
- Event Store → Fault Remediation: provides events
- Fault Quarantine → Kubernetes API: cordons node
- Node Drainer → Kubernetes API: drains workloads
- Fault Remediation → Janitor Controllers: requests repair
- Janitor Controllers → GPU Reset: reconciles reset (evidence: `janitor/main.go`)
- Janitor Controllers → CSP Provider: requests reboot
- CSP Provider → Cloud Provider: signals repair (evidence: `janitor-provider/main.go`)
- CSP Provider → Kubernetes API: reads node (evidence: `janitor-provider/main.go`)
- Lifecycle Manager → Kubernetes API: validates nodes
- Kubernetes Workload → Preflight: runs readiness check
- Preflight → Gang Checks: coordinates checks (evidence: `preflight/main.go`)
- Gang Checks → Kubernetes API: reads pod state (evidence: `preflight/pkg/controller/gang_controller.go`)