# slinkyproject/slurm-operator architecture

> Architecture diagram of the GitHub repository slinkyproject/slurm-operator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/slinkyproject/slurm-operator
- Repository: https://github.com/slinkyproject/slurm-operator

## Overview

##### Project Type & Purpose
- This is a Kubernetes Operator (written in Go) for managing Slurm clusters on Kubernetes. It defines Custom Resource Definitions (CRDs) for Cluster and NodeSet resources, runs a controller-manager and webhook server to reconcile those CRDs, and ships Helm charts to deploy both the operator itself and reference Slurm clusters.

##### File Structure Analysis
- api/v1alpha1: Go types + autogenerated deepcopy methods for Slinky Cluster and NodeSet CRDs.
- cmd/manager & cmd/webhook: entrypoints for the operator’s controller-manager (watches CRs, runs reconcilers) and the validating/mutation webhook.
- internal/controller: two reconcilers (cluster and nodeset). Each implements the Kubernetes controller-runtime pattern:
- cluster_controller.go handles Cluster CR—provisioning Slurm control plane
- nodeset_controller.go handles NodeSet CR—scaling and lifecycle of compute nodes
- config/: Kubernetes manifests and Kustomize overlays for CRDs, RBAC, webhook, manager Deployment, Prometheus monitors, cert-manager integration.
- helm/: two charts
- slurm-operator: packages Deployment of manager + webhook, CRDs, RBAC, ServiceAccount, and installation instructions
- slurm: reference implementation of a Slurm cluster (statefulsets for slurmdbd, controller, compute, login, REST API, secrets, ConfigMaps, Jobs for tokens)

##### Key Architectural Patterns
- Kubernetes Operator pattern (CRD + controller-runtime reconcilers + webhook)
- Reconciliation loop for desired vs actual state (create/update/delete Kubernetes primitives)
- Separation of concerns: webhook server (validation/mutation) vs manager (business logic)
- Helm charts for packaging and deploying both operator and Slurm cluster resources

##### Main Components
A. Kubernetes Control Plane
- API Server + etcd (CRD registry, built-in controllers)
B. Operator
- Manager Deployment
- Watches Cluster & NodeSet CRs via controller-runtime
- cluster-controller & nodeset-controller reconcile loops
- Webhook Deployment
- Mutating/validating Admission Webhooks for CR admission
- RBAC & ServiceAccount
- cert-manager integration & TLS certs for webhooks
- PrometheusServiceMonitor for metrics
C. Custom Resources
- Cluster CRD (slinky.slurm.net/Clusters)
- NodeSet CRD (slinky.slurm.net/NodeSets)
D. Helm‐deployed Slurm Cluster
- StatefulSet: slurmdbd (DB)
- StatefulSet: slurmctld (Controller)
- StatefulSet: compute NodeSets (slurmd)
- Deployment: login service
- Deployment: REST API
- ConfigMaps & Secrets (slurm.conf, JWT, credentials)
- Jobs for initial tokens & slurmctl commands
- Services for intra-cluster networking
E. External Integrations
- Helm (CLI/CI pipelines deploy operator and clusters)
- cert-manager (issues TLS certs for webhook)
- Prometheus (scrapes operator metrics)

##### Relationships & Interactions
- Operator Manager → Watches Cluster & NodeSet CRs → Calls Kubernetes API → Creates/updates HelmRelease‐style resources (StatefulSets, Deployments, Jobs, Services, ConfigMaps)
- Webhook → Intercepts CR create/update requests → Validates/mutates CR specs → Forwards to API Server
- Slurm Cluster Pods ↔ (TCP) each other (slurmd → slurmctld → slurmdbd)
- Prometheus → Scrapes metrics endpoint on manager Deployment
- cert-manager → Provides TLS certs mounted into webhook Pod

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Kubernetes Control Plane"
        APIServer["API Server"]:::controlplane
        Etcd["etcd"]:::controlplane
    end
    APIServer <---> Etcd

    subgraph "Operator Namespace"
        subgraph "Manager Deployment"
            MC["Cluster Reconciler"]:::operator
            MN["NodeSet Reconciler"]:::operator
        end
        Webhook["Webhook Deployment"]:::operator
        RBACSA["RBAC & ServiceAccount"]:::operator
        PromSM["Prometheus ServiceMonitor"]:::operator
    end

    subgraph "Custom Resources"
        ClusterCRD["Cluster CRD"]:::crd
        NodeSetCRD["NodeSet CRD"]:::crd
    end

    subgraph "Slurm Cluster Namespace"
        subgraph StatefulSets
            DBD["slurmdbd"]:::slurm
            CTL["slurmctld"]:::slurm
            ComputeSS["Compute NodeSets"]:::slurm
        end
        subgraph Deployments
            LoginDep["Login Service"]:::slurm
            RESTAPI["REST API"]:::slurm
        end
        ConfigMap["ConfigMaps & Secrets"]:::slurm
        Jobs["Init Jobs"]:::slurm
        Services["Services"]:::slurm
    end

    subgraph "External Integrations"
        HelmCLI["Helm CLI/CI"]:::external
        CertMgr["cert-manager"]:::external
        Prom["Prometheus"]:::external
    end

    HelmCLI -->|"install operator charts"| APIServer
    APIServer -->|"serves CRDs"| ClusterCRD
    APIServer -->|"serves CRDs"| NodeSetCRD

    ClusterCRD -->|"GET Cluster CR"| MC
    NodeSetCRD -->|"GET NodeSet CR"| MN

    MC -->|"GET/Watch CR"| APIServer
    MN -->|"GET/Watch CR"| APIServer

    MC -.->|"create StatefulSet"| DBD
    MC -.->|"create StatefulSet"| CTL
    MC -.->|"create Deployment"| LoginDep
    MC -.->|"create Deployment"| RESTAPI
    MC -.->|"create ConfigMap"| ConfigMap
    MC -.->|"create Job"| Jobs
    MC -.->|"create Service"| Services

    MN -.->|"scale Nodes"| ComputeSS

    APIServer -->|"ValidatingAdmissionReview"| Webhook
    Webhook -.->|"ValidationResponse"| APIServer

    Prom -->|"scrape /metrics"| MC
    CertMgr -->|"issues TLS certs"| Webhook

    click MC "https://github.com/slinkyproject/slurm-operator/blob/main/internal/controller/cluster/cluster_controller.go"
    click MN "https://github.com/slinkyproject/slurm-operator/blob/main/internal/controller/nodeset/nodeset_controller.go"
    click ClusterCRD "https://github.com/slinkyproject/slurm-operator/blob/main/api/v1alpha1/cluster_types.go"
    click NodeSetCRD "https://github.com/slinkyproject/slurm-operator/blob/main/api/v1alpha1/nodeset_types.go"
    click Manager "https://github.com/slinkyproject/slurm-operator/blob/main/cmd/manager/main.go"
    click Webhook "https://github.com/slinkyproject/slurm-operator/blob/main/cmd/webhook/main.go"
    click PromSM "https://github.com/slinkyproject/slurm-operator/blob/main/config/prometheus/monitor.yaml"
    click HelmCLI "https://github.com/slinkyproject/slurm-operator/tree/main/helm/slurm-operator/"

    classDef operator fill:#D0E6FF,stroke:#0366d6,color:#0366d6
    classDef controlplane fill:#e0ffdf,stroke:#259b24,color:#259b24
    classDef crd fill:#FFF4CC,stroke:#FFB300,color:#FFB300
    classDef slurm fill:#FFE0B2,stroke:#FB8C00,color:#E65100
    classDef external fill:#EEEEEE,stroke:#888888,color:#555555
```
