How slinkyproject/slurm-operator works
Project Type & Purpose
• This is a Kubernetes Operator (written in Go) for managing Slurm clusters on Kubernetes. It defines Custom Resource Definitions (CRDs) for Cluster and NodeSet resources, runs a controller-manager and webhook server to reconcile those CRDs, and ships Helm charts to deploy both the operator itself and reference Slurm clusters.
File Structure Analysis
• api/v1alpha1: Go types + autogenerated deepcopy methods for Slinky Cluster and NodeSet CRDs.
• cmd/manager & cmd/webhook: entrypoints for the operator’s controller-manager (watches CRs, runs reconcilers) and the validating/mutation webhook.
• internal/controller: two reconcilers (cluster and nodeset). Each implements the Kubernetes controller-runtime pattern:
• cluster_controller.go handles Cluster CR—provisioning Slurm control plane
• nodeset_controller.go handles NodeSet CR—scaling and lifecycle of compute nodes
• config/: Kubernetes manifests and Kustomize overlays for CRDs, RBAC, webhook, manager Deployment, Prometheus monitors, cert-manager integration.
• helm/: two charts
• slurm-operator: packages Deployment of manager + webhook, CRDs, RBAC, ServiceAccount, and installation instructions
• slurm: reference implementation of a Slurm cluster (statefulsets for slurmdbd, controller, compute, login, REST API, secrets, ConfigMaps, Jobs for tokens)
Key Architectural Patterns
• Kubernetes Operator pattern (CRD + controller-runtime reconcilers + webhook)
• Reconciliation loop for desired vs actual state (create/update/delete Kubernetes primitives)
• Separation of concerns: webhook server (validation/mutation) vs manager (business logic)
• Helm charts for packaging and deploying both operator and Slurm cluster resources
Main Components
A. Kubernetes Control Plane
• API Server + etcd (CRD registry, built-in controllers)
B. Operator
• Manager Deployment
• Watches Cluster & NodeSet CRs via controller-runtime
• cluster-controller & nodeset-controller reconcile loops
• Webhook Deployment
• Mutating/validating Admission Webhooks for CR admission
• RBAC & ServiceAccount
• cert-manager integration & TLS certs for webhooks
• PrometheusServiceMonitor for metrics
C. Custom Resources
• Cluster CRD (slinky.slurm.net/Clusters)
• NodeSet CRD (slinky.slurm.net/NodeSets)
D. Helm‐deployed Slurm Cluster
• StatefulSet: slurmdbd (DB)
• StatefulSet: slurmctld (Controller)
• StatefulSet: compute NodeSets (slurmd)
• Deployment: login service
• Deployment: REST API
• ConfigMaps & Secrets (slurm.conf, JWT, credentials)
• Jobs for initial tokens & slurmctl commands
• Services for intra-cluster networking
E. External Integrations
• Helm (CLI/CI pipelines deploy operator and clusters)
• cert-manager (issues TLS certs for webhook)
• Prometheus (scrapes operator metrics)
Relationships & Interactions
• Operator Manager → Watches Cluster & NodeSet CRs → Calls Kubernetes API → Creates/updates HelmRelease‐style resources (StatefulSets, Deployments, Jobs, Services, ConfigMaps)
• Webhook → Intercepts CR create/update requests → Validates/mutates CR specs → Forwards to API Server
• Slurm Cluster Pods ↔ (TCP) each other (slurmd → slurmctld → slurmdbd)
• Prometheus → Scrapes metrics endpoint on manager Deployment
• cert-manager → Provides TLS certs mounted into webhook Pod