Installation Guide

Table of Contents

Overview

Installation instructions for the Slurm Operator on Kubernetes.

Slurm Operator And CRDs

Install the cert-manager with its CRDs, if not already installed:

helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --namespace cert-manager --create-namespace \
  --set crds.enabled=true

Install the slurm-operator and its CRDs:

helm install slurm-operator-crds oci://ghcr.io/slinkyproject/charts/slurm-operator-crds
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --namespace=slinky --create-namespace

Check if the slurm-operator deployed successfully:

$ kubectl --namespace=slinky get pods --selector='app.kubernetes.io/instance=slurm-operator'
NAME                                      READY   STATUS    RESTARTS   AGE
slurm-operator-5d86d75979-6wflf           1/1     Running   0          1m
slurm-operator-webhook-567c84547b-kr7zq   1/1     Running   0          1m

Namespace-Scoped Watching

By default, the operator and webhook watch resources across all namespaces. To restrict them to specific namespaces, set the namespaces value to a comma-separated list:

helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'operator.namespaces=slurm-system\,production' \
  --set 'webhook.namespaces=slurm-system\,production' \
  --namespace=slinky --create-namespace

Note

When namespace scoping is enabled, the operator and webhook will only reconcile resources in the listed namespaces. Cluster-scoped resources (e.g. Nodes) are always watched regardless of this setting.

With CRDs As Subchart

If you intend to manage the slurm-operator and the CRDs in the same helm release, install it with the --set 'crds.enabled=true' argument.

helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'crds.enabled=true' \
  --namespace=slinky --create-namespace

Without cert-manager

If the cert-manager is not installed, then install the chart with the --set 'certManager.enabled=false' argument, to avoid signing certificates via cert-manager.

helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'certManager.enabled=false' \
  --namespace=slinky --create-namespace

Without cert-manager, the chart generates a self-signed CA and serving certificate via Helm’s genCA / genSignedCert functions at render time.

With an Externally-Managed Webhook Certificate

If your organization issues TLS certificates from its own PKI (HashiCorp Vault PKI, AWS Private CA, an internal CA, external-secrets, etc.), you can supply the webhook serving certificate as a pre-existing kubernetes.io/tls Secret. The chart will neither generate a certificate nor render any cert-manager resources, and rotation becomes the responsibility of your PKI tooling.

Create the Secret in the release namespace before installing the chart. It must contain tls.crt, tls.key, and ca.crt, and the certificate’s SANs must include both <webhook-service>.<namespace> and <webhook-service>.<namespace>.svc (default service name: slurm-operator-webhook).

kubectl create namespace slinky
kubectl create secret tls slurm-operator-webhook-ca \
  --namespace=slinky \
  --cert=tls.crt --key=tls.key
kubectl patch secret slurm-operator-webhook-ca \
  --namespace=slinky \
  --type=merge \
  -p "{\"data\":{\"ca.crt\":\"$(base64 < ca.crt | tr -d '\n')\"}}"

Then install the chart with certManager.enabled=false and externalCertInjection.enabled=true:

helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'certManager.enabled=false' \
  --set 'externalCertInjection.enabled=true' \
  --set 'externalCertInjection.secretName=slurm-operator-webhook-ca' \
  --namespace=slinky --create-namespace

The chart reads ca.crt from the Secret at install/upgrade time via Helm lookup and inlines it into every webhook’s clientConfig.caBundle. When you rotate the certificate, run helm upgrade to refresh the caBundle.

Warning

certManager.enabled=true and externalCertInjection.enabled=true are mutually exclusive (both would manage the same Secret). The chart will fail at template time if you set both.

Important

Offline rendering with helm template or helm install --dry-run=client cannot read the existing Secret through lookup, so this external-certificate configuration fails with “Secret … was not found” even when the Secret exists. Use helm template --dry-run=server or helm install --dry-run=server with access to the cluster containing the Secret. The cainjector annotations below do not bypass the lookup.

Note

When migrating from certManager.enabled=true, give the BYO Secret a name distinct from the chart-managed one. Reusing the cert-manager Secret name is supported but risky: if the BYO Secret has not yet been created on a fresh install, the Pod silently mounts the stale cert-manager Secret. The chart cannot detect this; the distinct-name workflow above avoids the race entirely.

If you would rather have cert-manager’s cainjector populate caBundle automatically from a Secret your PKI keeps refreshed (so helm upgrade is not needed for rotation), combine externalCertInjection with the webhook annotation pass-through. externalCertInjection makes the webhook pod mount the BYO Secret; the annotations make cainjector keep caBundle in sync with the same Secret’s ca.crt.

Important

For cainjector to read a ca.crt from a Secret (the cert-manager.io/inject-ca-from-secret annotation), the Secret MUST carry the annotation cert-manager.io/allow-direct-injection: "true". Without it, cainjector silently refuses to inject and the chart-rendered caBundle becomes the only source of truth. Add the annotation to your BYO Secret (either at create time or via kubectl annotate) before installing the chart.

kubectl create namespace slinky
kubectl create secret tls slurm-operator-webhook-byo \
  --namespace=slinky \
  --cert=tls.crt --key=tls.key
kubectl patch secret slurm-operator-webhook-byo \
  --namespace=slinky \
  --type=merge \
  -p "{\"data\":{\"ca.crt\":\"$(base64 < ca.crt | tr -d '\n')\"}}"
kubectl annotate secret slurm-operator-webhook-byo \
  --namespace=slinky \
  cert-manager.io/allow-direct-injection=true

helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
  --set 'certManager.enabled=false' \
  --set 'externalCertInjection.enabled=true' \
  --set 'externalCertInjection.secretName=slurm-operator-webhook-byo' \
  --set 'webhook.validatingAnnotations.cert-manager\.io/inject-ca-from-secret=slinky/slurm-operator-webhook-byo' \
  --set 'webhook.mutatingAnnotations.cert-manager\.io/inject-ca-from-secret=slinky/slurm-operator-webhook-byo' \
  --namespace=slinky --create-namespace

This requires cert-manager (specifically cainjector) to be running in the cluster, but the chart still does not manage the certificate itself. The chart-rendered caBundle (read from the Secret via lookup at install time) and the cainjector-written caBundle will be identical on install; on rotation, cainjector keeps caBundle correct without a helm upgrade.

Important

Do not omit externalCertInjection.enabled=true and rely on the annotations alone. Without it, the chart falls back to the self-signed genCA mode and the webhook pod serves a chart-generated certificate whose CA your PKI Secret does not match — the user’s PKI is silently ignored regardless of the cainjector annotation.

Slurm Cluster

Install a Slurm cluster via helm chart:

helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set-json 'nodesets={"slinky":{}}' \
  --set "partitions.all.enabled=true" \
  --namespace=slurm --create-namespace

Check if the Slurm cluster deployed successfully:

$ kubectl --namespace=slurm get pods
NAME                                  READY   STATUS    RESTARTS   AGE
slurm-controller-0                    2/2     Running   0          2m
slurm-restapi-77b9f969f7-kh4r8        1/1     Running   0          2m
slurm-worker-slinky-0                 2/2     Running   0          2m

Note

The quick-start values create a controller, REST API, and one NodeSet pod. Accounting and LoginSets appear only when separately enabled and configured.

Controller Persistence

By default, the Slurm controller (slurmctld) pod will store its state save data to a Persistent Volume (PV). Its Persistent Volume Claim (PVC) requests the Kubernetes default Storage Class.

If a default storage class is not defined or a specific storage class is desired, then you can install Slurm with the --set "controller.persistence.storageClassName=$STORAGE_CLASS" argument, where $STORAGE_CLASS matches an existing storage class.

kubectl get storageclasses.storage.k8s.io
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set "controller.persistence.storageClassName=$STORAGE_CLASS" \
  --namespace=slurm --create-namespace

Note

Deleting a PVC follows the PV’s reclaim policy. With Delete (the default for dynamically provisioned volumes), the PV and backing storage are deleted. With Retain, the storage remains and requires manual reclamation.

If Slurm controller (slurmctld) persistence is not desired (typically for testing), it can be disabled by installing Slurm with the --set 'controller.persistence.enabled=false' argument.

helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'controller.persistence.enabled=false' \
  --namespace=slurm --create-namespace

Warning

Without Slurm controller persistence, the state of the Slurm cluster is lost between Controller pod restarts. Moreover, these restarts may impact operation of the cluster and running workloads. Hence, disabling persistence is not recommended for production usage.

With Accounting

You will need to configure Slurm accounting to point at a database. There are multiple methods to provide a database for Slurm.

Either use:

Mariadb (Community Edition)

If you intend to enable accounting, install the mariadb-operator and its CRDs, if not already installed:

helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator
helm repo update
helm install mariadb-operator-crds mariadb-operator/mariadb-operator-crds
helm install mariadb-operator mariadb-operator/mariadb-operator \
  --namespace mariadb --create-namespace

Create the slurm namespace.

kubectl create namespace slurm

Create a mariadb database via CR.

kubectl apply -f - <<EOF
apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
  name: mariadb
  namespace: slurm
spec:
  rootPasswordSecretKeyRef:
    name: mariadb-root
    key: password
    generate: true
  username: slurm
  database: slurm_acct_db
  passwordSecretKeyRef:
    name: mariadb-password
    key: password
    generate: true
  storage:
    size: 16Gi
  myCnf: |
    [mariadb]
    bind-address=*
    default_storage_engine=InnoDB
    binlog_format=row
    innodb_autoinc_lock_mode=2
    innodb_buffer_pool_size=4096M
    innodb_lock_wait_timeout=900
    innodb_log_file_size=1024M
    max_allowed_packet=256M
EOF

Note

The mariadb database example above aligns with the Slurm chart’s default accounting.storageConfig. If your actual database configuration is different, then you will have to update the accounting.storageConfig to work with your configuration.

Then install a Slurm cluster via helm chart with the --set 'accounting.enabled=true' argument.

helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'accounting.enabled=true' \
  --namespace=slurm --create-namespace

With Metrics

If you intend to collect metrics, install prometheus and its CRDs, if not already installed:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack \
  --set 'installCRDs=true' \
  --namespace prometheus --create-namespace

Then enable Slurm metrics and the Prometheus service monitor. The release label matches the default ServiceMonitor selector of the prometheus release installed above.

helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'controller.metrics.enabled=true' \
  --set 'controller.metrics.serviceMonitor.enabled=true' \
  --set 'controller.metrics.serviceMonitor.labels.release=prometheus' \
  --namespace=slurm --create-namespace

With Login

You will need to configure the Slurm chart such that the login pods can communicate with an identity service via sssd.

Warning

In this example, you will need to supply an sssd.conf (at ${HOME}/sssd.conf) that is configured for your environment. Mount it at /etc/sssd/sssd.conf with mode 0600. sssd will not start if the file is readable by anyone other than root.

Create a Secret from that file, then install a Slurm cluster via helm chart. The LoginSet mounts the Secret into the login container:

kubectl create namespace slurm
kubectl create secret generic sssd-conf \
  --namespace=slurm \
  --from-file=sssd.conf="${HOME}/sssd.conf"

cat > login-values.yaml <<EOF
loginsets:
  slinky:
    enabled: true
    login:
      volumeMounts:
        - name: sssd-conf
          mountPath: /etc/sssd/sssd.conf
          subPath: sssd.conf
          readOnly: true
    podSpec:
      volumes:
        - name: sssd-conf
          secret:
            secretName: sssd-conf
            defaultMode: 0600
EOF

helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --values login-values.yaml \
  --namespace=slurm

With root Authorized Keys

Note

Even if sssd is misconfigured, this method can still be used to SSH into the pod.

Install a Slurm cluster via helm chart with the --set 'loginsets.slinky.enabled=true' and --set-file "loginsets.slinky.rootSshAuthorizedKeys=${HOME}/.ssh/id_ed25519.pub" arguments.

helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
  --set 'loginsets.slinky.enabled=true' \
  --set-file "loginsets.slinky.rootSshAuthorizedKeys=${HOME}/.ssh/id_ed25519.pub" \
  --namespace=slurm --create-namespace

Testing Slurm

SSH through the login service:

SLURM_LOGIN_HOST="$(kubectl get services -n slurm slurm-login-slinky -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')"
SLURM_LOGIN_PORT="$(kubectl get services -n slurm slurm-login-slinky -o jsonpath='{.spec.ports[0].port}')"
## Assuming your public SSH key was configured in `loginsets.slinky.rootSshAuthorizedKeys`.
ssh -p ${SLURM_LOGIN_PORT:-22} root@${SLURM_LOGIN_HOST}
## Assuming SSSD was configured correctly.
ssh -p ${SLURM_LOGIN_PORT:-22} ${USER}@${SLURM_LOGIN_HOST}

Then, from a login pod, run Slurm commands to quickly test that Slurm is functioning:

sinfo
srun hostname
sbatch --wrap="sleep 60"
squeue
sacct

See Slurm Commands for more details on how to interact with Slurm.

With GPUs

The following describes how to make GPUs present on a Kubernetes cluster available within Slurm when using Slurm-operator.

The gres.conf must have GRES defined for each node with GPUs. For dynamic GRES detection, it is recommended to use AutoDetect. The following example uses dynamic GRES with NVIDIA GPUs.

configFiles:
  gres.conf: |
    AutoDetect=nvidia

Slurm requires that GresTypes contains the “gpu” resource. Set the value in controller.extraConf or controller.extraConfMap.

controller:
  extraConfMap:
    GresTypes: "gpu"

NodeSets should request GPUs in accordance with device plugins or DRA. In addition, extraConf or extraConfMap needs to define a GRES in accordance with the GPUs it should be allocated to.

The following is an example of a gpu-gb200 NodeSet which has 4 GB200 GPUs. This example assumes that the NVIDIA gpu-operator is running on the Kubernetes cluster.

nodesets:
  gpu-gb200:
    partition:
      enabled: true
    slurmd:
      resources:
        limits:
          nvidia.com/gpu: 4
    extraConfMap:
      Gres: ["gpu:GB200:4"]

Within the NodeSet pod, the allocated GPUs are visible in the output of the nvidia-smi command:

$ kubectl exec -n slurm slurm-controller-0 -- srun --partition=gpu-gb200 --gres=gpu:GB200:4 bash -c 'echo GPU Devices Available on $(hostname):; nvidia-smi'
GPU Devices Available on gpu-gb200-0:

Mon Apr 20 17:03:28 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB200                   On  |   00000008:01:00.0 Off |                    0 |
| N/A   37C    P0            166W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GB200                   On  |   00000009:01:00.0 Off |                    0 |
| N/A   37C    P0            157W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GB200                   On  |   00000018:01:00.0 Off |                    0 |
| N/A   36C    P0            150W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GB200                   On  |   00000019:01:00.0 Off |                    0 |
| N/A   36C    P0            172W / 1200W |       0MiB / 189471MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

With IMEX

NVIDIA GB200 & GB300 NVL72 systems provide the NVIDIA Internode Memory Exchange/Management Service (IMEX) service. IMEX facilitates shared memory operations across nodes on an NVLink Fabric. IMEX is crucial to taking full advantage of the NVL72 rackscale systems.

Slurm-operator supports the use of IMEX channels, in order to provide memory isolation within an IMEX domain. To use IMEX on Slurm-operator, the DRA Driver NVIDIA GPU should be installed in order to automate the management and allocation of the IMEX daemons. This can be installed using the NVIDIA GPU Operator.

Historically, baremetal implementations of IMEX domain management with Slurm made use of complicated prolog and epilog scripts to stand up IMEX channels on nodes prior to job launch, and to clean them up after job completion. With Slurm-operator, these scripts should not be used, as they may interfere with the operations of the GPU DRA driver.

Limitations

As Slurm-operator relies on DRA to configure IMEX channels on the underlying Kubernetes nodes, Slurm-operator does not support per-job configuration of IMEX channels. If this is a hard requirement for a site, the ComputeDomain that is claimed by a NodeSet should contain multiple channels, and prolog and epilog scripts should be used to limit user access to these channels using Linux permissions or mounts.

Configuration

Below is a simple ComputeDomain, which injects all channels into the NodeSet’s slurmd pod:

---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: slurm-compute-domain
  namespace: slurm
spec:
  channel:
    allocationMode: All
    resourceClaimTemplate:
      name: slurm-test-compute-domain
  numNodes: 0

The SwitchType field in slurm.conf should be set so that slurmctld can identify the type of switch that is being used for application communications. In the Slurm chart, set it with controller.extraConfMap; the chart renders the result into the Controller’s spec.extraConf string:

controller:
  extraConfMap:
    SwitchType: "switch/nvidia_imex"
    SwitchParameters: ""

Resource limits and claims for Slinky NodeSets can be specified in the spec.slurmd.resources field of the CRD or the Slurm Helm chart:

nodesets:
  slinky:
    slurmd:
      resources:
        limits:
          nvidia.com/gpu: 4
        claims:
          - name: compute-domain-channel

DRA ResourceClaims can be specified for Slinky NodeSets in the Helm chart values:

nodesets:
  slinky:
    podSpec:
      resourceClaims:
        - name: compute-domain-channel
          resourceClaimTemplateName: slurm-test-compute-domain

Once the Slurm chart has been deployed, a ResourceClaim should be created for the NodeSet pod:

$ kubectl get resourceclaim -n slurm
NAME                                                     STATE                AGE
slurm-worker-slinky-jvb8l-compute-domain-channel-4rpsc   allocated,reserved   7m8s

A simple job can be launched to confirm successful creation of the IMEX channels on the NodeSet pod:

$ kubectl exec -n slurm slurm-controller-0 -- srun bash -c \ "printf '\n\n'; echo === IMEX Channel Test ===; \
 echo Job \$SLURM_JOB_ID Node \$SLURMD_NODENAME; \
echo Channel devices:; \
ls /dev/nvidia-caps-imex-channels/ | grep channel || echo No channels found; \
ls -la /dev/nvidia-caps-imex-channels/channel* 2>/dev/null || echo No channel devices;"


=== IMEX Channel Test ===
Job 74 Node validation-k8s-gpu-02
Channel devices:
channel1
crw-rw-rw- 1 root root 509, 1 Apr 20 17:08 /dev/nvidia-caps-imex-channels/channel1