Installation Guide
Table of Contents
Overview
Installation instructions for the Slurm Operator on Kubernetes.
Slurm Operator And CRDs
Install the cert-manager with its CRDs, if not already installed:
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--namespace cert-manager --create-namespace \
--set crds.enabled=true
Install the slurm-operator and its CRDs:
helm install slurm-operator-crds oci://ghcr.io/slinkyproject/charts/slurm-operator-crds
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
--namespace=slinky --create-namespace
Check if the slurm-operator deployed successfully:
$ kubectl --namespace=slinky get pods --selector='app.kubernetes.io/instance=slurm-operator'
NAME READY STATUS RESTARTS AGE
slurm-operator-5d86d75979-6wflf 1/1 Running 0 1m
slurm-operator-webhook-567c84547b-kr7zq 1/1 Running 0 1m
Namespace-Scoped Watching
By default, the operator and webhook watch resources across all namespaces. To
restrict them to specific namespaces, set the namespaces value to a
comma-separated list:
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
--set 'operator.namespaces=slurm-system\,production' \
--set 'webhook.namespaces=slurm-system\,production' \
--namespace=slinky --create-namespace
Note
When namespace scoping is enabled, the operator and webhook will only reconcile resources in the listed namespaces. Cluster-scoped resources (e.g. Nodes) are always watched regardless of this setting.
With CRDs As Subchart
If you intend to manage the slurm-operator and the CRDs in the same helm
release, install it with the --set 'crds.enabled=true' argument.
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
--set 'crds.enabled=true' \
--namespace=slinky --create-namespace
Without cert-manager
If the cert-manager is not installed, then install the chart with the
--set 'certManager.enabled=false' argument, to avoid signing certificates via
cert-manager.
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
--set 'certManager.enabled=false' \
--namespace=slinky --create-namespace
Without cert-manager, the chart generates a self-signed CA and serving
certificate via Helm’s genCA / genSignedCert functions at render time.
With an Externally-Managed Webhook Certificate
If your organization issues TLS certificates from its own PKI (HashiCorp Vault
PKI, AWS Private CA, an internal CA, external-secrets, etc.), you can supply
the webhook serving certificate as a pre-existing kubernetes.io/tls Secret.
The chart will neither generate a certificate nor render any cert-manager
resources, and rotation becomes the responsibility of your PKI tooling.
Create the Secret in the release namespace before installing the chart. It must
contain tls.crt, tls.key, and ca.crt, and the certificate’s SANs must
include both <webhook-service>.<namespace> and
<webhook-service>.<namespace>.svc (default service name:
slurm-operator-webhook).
kubectl create namespace slinky
kubectl create secret tls slurm-operator-webhook-ca \
--namespace=slinky \
--cert=tls.crt --key=tls.key
kubectl patch secret slurm-operator-webhook-ca \
--namespace=slinky \
--type=merge \
-p "{\"data\":{\"ca.crt\":\"$(base64 < ca.crt | tr -d '\n')\"}}"
Then install the chart with certManager.enabled=false and
externalCertInjection.enabled=true:
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
--set 'certManager.enabled=false' \
--set 'externalCertInjection.enabled=true' \
--set 'externalCertInjection.secretName=slurm-operator-webhook-ca' \
--namespace=slinky --create-namespace
The chart reads ca.crt from the Secret at install/upgrade time via Helm
lookup and inlines it into every webhook’s clientConfig.caBundle. When you
rotate the certificate, run helm upgrade to refresh the caBundle.
Warning
certManager.enabled=true and externalCertInjection.enabled=true are
mutually exclusive (both would manage the same Secret). The chart will fail at
template time if you set both.
Important
Offline rendering with helm template or helm install --dry-run=client
cannot read the existing Secret through lookup, so this external-certificate
configuration fails with “Secret … was not found” even when the Secret exists.
Use helm template --dry-run=server or helm install --dry-run=server with
access to the cluster containing the Secret. The cainjector annotations below
do not bypass the lookup.
Note
When migrating from certManager.enabled=true, give the BYO Secret a name
distinct from the chart-managed one. Reusing the cert-manager Secret name is
supported but risky: if the BYO Secret has not yet been created on a fresh
install, the Pod silently mounts the stale cert-manager Secret. The chart
cannot detect this; the distinct-name workflow above avoids the race entirely.
If you would rather have cert-manager’s cainjector populate caBundle
automatically from a Secret your PKI keeps refreshed (so helm upgrade is not
needed for rotation), combine externalCertInjection with the webhook
annotation pass-through. externalCertInjection makes the webhook pod mount the
BYO Secret; the annotations make cainjector keep caBundle in sync with the
same Secret’s ca.crt.
Important
For cainjector to read a ca.crt from a Secret (the
cert-manager.io/inject-ca-from-secret annotation), the Secret MUST carry the
annotation cert-manager.io/allow-direct-injection: "true". Without it,
cainjector silently refuses to inject and the chart-rendered caBundle
becomes the only source of truth. Add the annotation to your BYO Secret
(either at create time or via kubectl annotate) before installing the chart.
kubectl create namespace slinky
kubectl create secret tls slurm-operator-webhook-byo \
--namespace=slinky \
--cert=tls.crt --key=tls.key
kubectl patch secret slurm-operator-webhook-byo \
--namespace=slinky \
--type=merge \
-p "{\"data\":{\"ca.crt\":\"$(base64 < ca.crt | tr -d '\n')\"}}"
kubectl annotate secret slurm-operator-webhook-byo \
--namespace=slinky \
cert-manager.io/allow-direct-injection=true
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \
--set 'certManager.enabled=false' \
--set 'externalCertInjection.enabled=true' \
--set 'externalCertInjection.secretName=slurm-operator-webhook-byo' \
--set 'webhook.validatingAnnotations.cert-manager\.io/inject-ca-from-secret=slinky/slurm-operator-webhook-byo' \
--set 'webhook.mutatingAnnotations.cert-manager\.io/inject-ca-from-secret=slinky/slurm-operator-webhook-byo' \
--namespace=slinky --create-namespace
This requires cert-manager (specifically cainjector) to be running in the
cluster, but the chart still does not manage the certificate itself. The
chart-rendered caBundle (read from the Secret via lookup at install time)
and the cainjector-written caBundle will be identical on install; on rotation,
cainjector keeps caBundle correct without a helm upgrade.
Important
Do not omit externalCertInjection.enabled=true and rely on the annotations
alone. Without it, the chart falls back to the self-signed genCA mode and
the webhook pod serves a chart-generated certificate whose CA your PKI Secret
does not match — the user’s PKI is silently ignored regardless of the
cainjector annotation.
Slurm Cluster
Install a Slurm cluster via helm chart:
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
--set-json 'nodesets={"slinky":{}}' \
--set "partitions.all.enabled=true" \
--namespace=slurm --create-namespace
Check if the Slurm cluster deployed successfully:
$ kubectl --namespace=slurm get pods
NAME READY STATUS RESTARTS AGE
slurm-controller-0 2/2 Running 0 2m
slurm-restapi-77b9f969f7-kh4r8 1/1 Running 0 2m
slurm-worker-slinky-0 2/2 Running 0 2m
Note
The quick-start values create a controller, REST API, and one NodeSet pod. Accounting and LoginSets appear only when separately enabled and configured.
Controller Persistence
By default, the Slurm controller (slurmctld) pod will store its state save data to a Persistent Volume (PV). Its Persistent Volume Claim (PVC) requests the Kubernetes default Storage Class.
If a default storage class is not defined or a specific storage class is
desired, then you can install Slurm with the
--set "controller.persistence.storageClassName=$STORAGE_CLASS" argument, where
$STORAGE_CLASS matches an existing storage class.
kubectl get storageclasses.storage.k8s.io
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
--set "controller.persistence.storageClassName=$STORAGE_CLASS" \
--namespace=slurm --create-namespace
Note
Deleting a PVC follows the PV’s reclaim policy. With
Delete (the default for dynamically provisioned volumes), the PV and backing
storage are deleted. With Retain, the storage remains and requires manual
reclamation.
If Slurm controller (slurmctld) persistence is not desired (typically for
testing), it can be disabled by installing Slurm with the
--set 'controller.persistence.enabled=false' argument.
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
--set 'controller.persistence.enabled=false' \
--namespace=slurm --create-namespace
Warning
Without Slurm controller persistence, the state of the Slurm cluster is lost between Controller pod restarts. Moreover, these restarts may impact operation of the cluster and running workloads. Hence, disabling persistence is not recommended for production usage.
With Accounting
You will need to configure Slurm accounting to point at a database. There are multiple methods to provide a database for Slurm.
Either use:
the mariadb-operator
the mysql-operator
any Slurm compatible database
mysql/mariadb compatible alternatives
managed cloud database service
Mariadb (Community Edition)
If you intend to enable accounting, install the mariadb-operator and its CRDs, if not already installed:
helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator
helm repo update
helm install mariadb-operator-crds mariadb-operator/mariadb-operator-crds
helm install mariadb-operator mariadb-operator/mariadb-operator \
--namespace mariadb --create-namespace
Create the slurm namespace.
kubectl create namespace slurm
Create a mariadb database via CR.
kubectl apply -f - <<EOF
apiVersion: k8s.mariadb.com/v1alpha1
kind: MariaDB
metadata:
name: mariadb
namespace: slurm
spec:
rootPasswordSecretKeyRef:
name: mariadb-root
key: password
generate: true
username: slurm
database: slurm_acct_db
passwordSecretKeyRef:
name: mariadb-password
key: password
generate: true
storage:
size: 16Gi
myCnf: |
[mariadb]
bind-address=*
default_storage_engine=InnoDB
binlog_format=row
innodb_autoinc_lock_mode=2
innodb_buffer_pool_size=4096M
innodb_lock_wait_timeout=900
innodb_log_file_size=1024M
max_allowed_packet=256M
EOF
Note
The mariadb database example above aligns with the Slurm chart’s default
accounting.storageConfig. If your actual database configuration is
different, then you will have to update the accounting.storageConfig to work
with your configuration.
Then install a Slurm cluster via helm chart with the
--set 'accounting.enabled=true' argument.
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
--set 'accounting.enabled=true' \
--namespace=slurm --create-namespace
With Metrics
If you intend to collect metrics, install prometheus and its CRDs, if not already installed:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus prometheus-community/kube-prometheus-stack \
--set 'installCRDs=true' \
--namespace prometheus --create-namespace
Then enable Slurm metrics and the Prometheus service monitor. The release
label matches the default ServiceMonitor selector of the prometheus release
installed above.
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
--set 'controller.metrics.enabled=true' \
--set 'controller.metrics.serviceMonitor.enabled=true' \
--set 'controller.metrics.serviceMonitor.labels.release=prometheus' \
--namespace=slurm --create-namespace
With Login
You will need to configure the Slurm chart such that the login pods can communicate with an identity service via sssd.
Warning
In this example, you will need to supply an sssd.conf (at
${HOME}/sssd.conf) that is configured for your environment. Mount it at
/etc/sssd/sssd.conf with mode 0600. sssd will not start if the file is
readable by anyone other than root.
Create a Secret from that file, then install a Slurm cluster via helm chart. The LoginSet mounts the Secret into the login container:
kubectl create namespace slurm
kubectl create secret generic sssd-conf \
--namespace=slurm \
--from-file=sssd.conf="${HOME}/sssd.conf"
cat > login-values.yaml <<EOF
loginsets:
slinky:
enabled: true
login:
volumeMounts:
- name: sssd-conf
mountPath: /etc/sssd/sssd.conf
subPath: sssd.conf
readOnly: true
podSpec:
volumes:
- name: sssd-conf
secret:
secretName: sssd-conf
defaultMode: 0600
EOF
helm install slurm oci://ghcr.io/slinkyproject/charts/slurm \
--values login-values.yaml \
--namespace=slurm
Testing Slurm
SSH through the login service:
SLURM_LOGIN_HOST="$(kubectl get services -n slurm slurm-login-slinky -o jsonpath='{.status.loadBalancer.ingress[0].ip}{.status.loadBalancer.ingress[0].hostname}')"
SLURM_LOGIN_PORT="$(kubectl get services -n slurm slurm-login-slinky -o jsonpath='{.spec.ports[0].port}')"
## Assuming your public SSH key was configured in `loginsets.slinky.rootSshAuthorizedKeys`.
ssh -p ${SLURM_LOGIN_PORT:-22} root@${SLURM_LOGIN_HOST}
## Assuming SSSD was configured correctly.
ssh -p ${SLURM_LOGIN_PORT:-22} ${USER}@${SLURM_LOGIN_HOST}
Then, from a login pod, run Slurm commands to quickly test that Slurm is functioning:
sinfo
srun hostname
sbatch --wrap="sleep 60"
squeue
sacct
See Slurm Commands for more details on how to interact with Slurm.
With GPUs
The following describes how to make GPUs present on a Kubernetes cluster available within Slurm when using Slurm-operator.
The gres.conf must have GRES defined for each node with GPUs. For dynamic
GRES detection, it is recommended to use AutoDetect. The following example
uses dynamic GRES with NVIDIA GPUs.
configFiles:
gres.conf: |
AutoDetect=nvidia
Slurm requires that GresTypes contains the “gpu” resource. Set the value in
controller.extraConf or controller.extraConfMap.
controller:
extraConfMap:
GresTypes: "gpu"
NodeSets should request GPUs in accordance with device plugins
or DRA. In addition, extraConf or extraConfMap needs to define a GRES in
accordance with the GPUs it should be allocated to.
The following is an example of a gpu-gb200 NodeSet which has 4 GB200 GPUs.
This example assumes that the NVIDIA gpu-operator is
running on the Kubernetes cluster.
nodesets:
gpu-gb200:
partition:
enabled: true
slurmd:
resources:
limits:
nvidia.com/gpu: 4
extraConfMap:
Gres: ["gpu:GB200:4"]
Within the NodeSet pod, the allocated GPUs are visible in the output of the
nvidia-smi command:
$ kubectl exec -n slurm slurm-controller-0 -- srun --partition=gpu-gb200 --gres=gpu:GB200:4 bash -c 'echo GPU Devices Available on $(hostname):; nvidia-smi'
GPU Devices Available on gpu-gb200-0:
Mon Apr 20 17:03:28 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01 Driver Version: 590.48.01 CUDA Version: 13.1 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GB200 On | 00000008:01:00.0 Off | 0 |
| N/A 37C P0 166W / 1200W | 0MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GB200 On | 00000009:01:00.0 Off | 0 |
| N/A 37C P0 157W / 1200W | 0MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 2 NVIDIA GB200 On | 00000018:01:00.0 Off | 0 |
| N/A 36C P0 150W / 1200W | 0MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
| 3 NVIDIA GB200 On | 00000019:01:00.0 Off | 0 |
| N/A 36C P0 172W / 1200W | 0MiB / 189471MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
With IMEX
NVIDIA GB200 & GB300 NVL72 systems provide the NVIDIA Internode Memory Exchange/Management Service (IMEX) service. IMEX facilitates shared memory operations across nodes on an NVLink Fabric. IMEX is crucial to taking full advantage of the NVL72 rackscale systems.
Slurm-operator supports the use of IMEX channels, in order to provide memory isolation within an IMEX domain. To use IMEX on Slurm-operator, the DRA Driver NVIDIA GPU should be installed in order to automate the management and allocation of the IMEX daemons. This can be installed using the NVIDIA GPU Operator.
Historically, baremetal implementations of IMEX domain management with Slurm made use of complicated prolog and epilog scripts to stand up IMEX channels on nodes prior to job launch, and to clean them up after job completion. With Slurm-operator, these scripts should not be used, as they may interfere with the operations of the GPU DRA driver.
Limitations
As Slurm-operator relies on DRA to configure IMEX channels on the underlying Kubernetes nodes, Slurm-operator does not support per-job configuration of IMEX channels. If this is a hard requirement for a site, the ComputeDomain that is claimed by a NodeSet should contain multiple channels, and prolog and epilog scripts should be used to limit user access to these channels using Linux permissions or mounts.
Configuration
Below is a simple ComputeDomain, which injects all channels into the NodeSet’s
slurmd pod:
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
name: slurm-compute-domain
namespace: slurm
spec:
channel:
allocationMode: All
resourceClaimTemplate:
name: slurm-test-compute-domain
numNodes: 0
The SwitchType field in slurm.conf should be set so that slurmctld can
identify the type of switch that is being used for application communications.
In the Slurm chart, set it with controller.extraConfMap; the chart renders the
result into the Controller’s spec.extraConf string:
controller:
extraConfMap:
SwitchType: "switch/nvidia_imex"
SwitchParameters: ""
Resource limits and claims for Slinky NodeSets can be specified in the
spec.slurmd.resources field of the CRD or the Slurm Helm chart:
nodesets:
slinky:
slurmd:
resources:
limits:
nvidia.com/gpu: 4
claims:
- name: compute-domain-channel
DRA ResourceClaims can be specified for Slinky NodeSets in the Helm chart values:
nodesets:
slinky:
podSpec:
resourceClaims:
- name: compute-domain-channel
resourceClaimTemplateName: slurm-test-compute-domain
Once the Slurm chart has been deployed, a ResourceClaim should be created for the NodeSet pod:
$ kubectl get resourceclaim -n slurm
NAME STATE AGE
slurm-worker-slinky-jvb8l-compute-domain-channel-4rpsc allocated,reserved 7m8s
A simple job can be launched to confirm successful creation of the IMEX channels on the NodeSet pod:
$ kubectl exec -n slurm slurm-controller-0 -- srun bash -c \ "printf '\n\n'; echo === IMEX Channel Test ===; \
echo Job \$SLURM_JOB_ID Node \$SLURMD_NODENAME; \
echo Channel devices:; \
ls /dev/nvidia-caps-imex-channels/ | grep channel || echo No channels found; \
ls -la /dev/nvidia-caps-imex-channels/channel* 2>/dev/null || echo No channel devices;"
=== IMEX Channel Test ===
Job 74 Node validation-k8s-gpu-02
Channel devices:
channel1
crw-rw-rw- 1 root root 509, 1 Apr 20 17:08 /dev/nvidia-caps-imex-channels/channel1