How nvidia/nvsentinel works
NVSentinel is a modular Kubernetes GPU-fault detection and repair platform. Its documented user-to-result path is monitoring faults, isolating affected nodes through cordon and optional drain, then optionally repairing them with a GPU reset or reboot; preflight adds an active readiness check before workloads start. The README describes modules coordinating through a shared MongoDB event store and Kubernetes API, not direct module-to-module calls. The architecture excerpt is truncated and most module internals were not sampled, so uncertain links are dashed and uncited rather than presented as inspected call paths.
Main entry points: postgresql_pipeline_builder.go Event Store · platform-connectors/main.go Platform Connectors · janitor/main.go Janitor Controllers · janitor-provider/main.go CSP Provider
Components 21
Fault Detection
- GPU Monitor
health-monitors/gpu-health-monitor/gpu_health_monitor/dcgm_watcher/dcgm.py - Syslog Monitor
health-monitors/syslog-health-monitor/main.go - NIC Monitor
health-monitors/nic-health-monitor/main.go - CSP Monitor
health-monitors/csp-health-monitor/cmd/csp-health-monitor/main.go - Object Monitor
health-monitors/kubernetes-object-monitor/main.go - Event Analyzer
health-events-analyzer/main.go
Event Ingestion
- Platform Connectors
platform-connectors/main.go - Event Exporter
event-exporter/main.go - Event Store
store-client/pkg/client/postgresql_pipeline_builder.go
Node Protection
- Fault Quarantine
fault-quarantine/main.go - Node Drainer
node-drainer/main.go
Remediation
- Fault Remediation
fault-remediation/main.go - Janitor Controllers
janitor/main.go - GPU Reset
janitor/pkg/controller/gpureset_controller.go - CSP Provider
janitor-provider/main.go - Lifecycle Manager
lifecycle-manager/main.go
Workload Readiness
- Preflight
preflight/main.go - Gang Checks
preflight/pkg/controller/gang_controller.go
Other components
- Kubernetes Workload
- Kubernetes API
- Cloud Provider
Connections 21
- GPU Monitor → Platform Connectors: publishes events
- Syslog Monitor → Platform Connectors: publishes events
- NIC Monitor → Platform Connectors: publishes events
- CSP Monitor → Event Store: reports maintenance
- Object Monitor → Event Store: reports faults
- Platform Connectors → Event Store: stores events
- Event Store → Event Analyzer: supplies events
- Event Store → Event Exporter: exports events
- Event Store → Fault Quarantine: provides events
- Event Store → Fault Remediation: provides events
- Fault Quarantine → Kubernetes API: cordons node
- Node Drainer → Kubernetes API: drains workloads
- Fault Remediation → Janitor Controllers: requests repair
- Janitor Controllers → GPU Reset: reconciles reset
janitor/main.go - Janitor Controllers → CSP Provider: requests reboot
- CSP Provider → Cloud Provider: signals repair
janitor-provider/main.go - CSP Provider → Kubernetes API: reads node
janitor-provider/main.go - Lifecycle Manager → Kubernetes API: validates nodes
- Kubernetes Workload → Preflight: runs readiness check
- Preflight → Gang Checks: coordinates checks
preflight/main.go - Gang Checks → Kubernetes API: reads pod state
preflight/pkg/controller/gang_controller.go