QuickStart Guide
Table of Contents
Overview
This quickstart guide will help you get the slurm-bridge running and configured with your existing Slurm cluster.
This document assumes a basic understanding of Kubernetes architecture. It is highly recommended that those who are unfamiliar with the core concepts of Kubernetes review the documentation on Kubernetes, pods, and nodes before getting started.
Pre-requisites
Kubernetes and Slurm versions supported by the compatibility matrix.
Kubernetes workers configured as external or hybrid nodes. Hybrid nodes run colocated kubelet and slurmd; external nodes do not require slurmd.
To install a native
slurmdon an existing Kubernetes worker, follow Host-installed Hybrid Slurm Nodes.
The built-in Workload and PodGroup APIs enabled on Kubernetes 1.36 or 1.37. Helm automatically disables native group scheduling on Kubernetes 1.35. See Built-in PodGroup for cluster configuration.
Matching NodeNames in Slurm and Kubernetes for all overlapping nodes
In the event that the colocated node’s Slurm NodeName does not match the Kubernetes Node name, you should patch the Kubernetes node with a label to allow
slurm-bridgeto map the colocated Kubernetes and Slurm node.kubectl patch node $KUBERNETES_NODENAME -p "{\"metadata\":{\"labels\":{\"slinky.slurm.net/slurm-nodename\":\"$SLURM_NODENAME\"}}}"
Installation
1. Install the required helm charts
helm install cert-manager --repo https://charts.jetstack.io cert-manager \
--namespace cert-manager --create-namespace --set crds.enabled=true
2. Create a secret for slurm-bridge
Create a secret for slurm-bridge to communicate with Slurm.
When running Slurm in slurm-operator:
kubectl apply -f - <<EOF
apiVersion: slinky.slurm.net/v1beta1
kind: Token
metadata:
name: slurm-bridge-token
namespace: slurm
spec:
jwtKeyRef:
name: slurm-auth-jwt
key: jwt.key
secretRef:
name: slurm-bridge-token
key: SLURM_JWT
username: slurm
refresh: true
lifetime: 8760h
EOF
When running Slurm on baremetal:
export $(scontrol token username=slurm lifespan=infinite)
kubectl create namespace slurm
kubectl create secret generic slurm-bridge-token --namespace=slurm --from-literal="SLURM_JWT=$SLURM_JWT" --type=Opaque
3. Download and configure values.yaml for the slurm-bridge helm chart
The helm chart used by slurm-bridge has a number of parameters in
values.yaml
that can be modified to tweak various parameters of slurm-bridge. Most of these
values should work without modification.
Download the chart defaults before editing:
helm show values oci://ghcr.io/slinkyproject/charts/slurm-bridge > values.yaml
Depending on your Slurm configuration, you may need to configure the following variables:
schedulerConfig.partition- this is the default partition with whichslurm-bridgewill associate jobs. This partition should only include nodes that map to Kubernetes workers, using either external or hybrid nodes. The default value of this variable isslurm-bridge.schedulerConfig.mcsLabel- sets the MCS category used whenever a workload requests non-exclusive placement. A non-empty label and the additional Slurm configuration are required, as described in Hybrid Workload Isolation.sharedConfig.slurmRestApi- the URL used byslurm-bridgeto interact with the Slurm REST API. Changing this value may be necessary if you run the REST API on a different URL or port. The default value of this variable ishttp://slurm-restapi.slurm:6820
4. Install the slurm-bridge Helm Chart
helm install slurm-bridge oci://ghcr.io/slinkyproject/charts/slurm-bridge \
--namespace=slurm --create-namespace --values values.yaml
kubectl create namespace slurm-bridge
Note
slurm-bridge must be able to communicate with Slurm REST API. By default, it
assumes a default Slurm chart installation and uses
http://slurm-restapi.slurm:6820.
You can check if your cluster deployed successfully with:
kubectl --namespace=slurm get pods
Your output should be similar to:
NAME READY STATUS RESTARTS AGE
slurm-bridge-admission-85f89cf884-8c9jt 1/1 Running 0 1m0s
slurm-bridge-scheduler-5484467f55-wtspk 1/1 Running 0 1m0s
Running Your First Job
slurm-bridge supports JobSet and PodGroup resources and their pods. If
your workload requires co-scheduled pod launch (e.g. MPI, multi-node), use a
Gang PodGroup. A JobSet alone uses one Slurm allocation per Pod; see
JobSets.
Now that slurm-bridge is configured, we can write a workload. slurm-bridge
schedules Kubernetes workloads using the Slurm scheduler by translating a
Kubernetes workload in the form of Jobs, JobSets, Pods,
and PodGroups into a representative Slurm job, which is used for
scheduling purposes. Once a workload is allocated resources, slurm-bridge
binds the Kubernetes workload to the allocated nodes and the Kubelet executes
it. There are
example workload
definitions in the slurm-bridge repo.
Here’s an example of a simple job, found in hack/examples/job/single.yaml:
---
apiVersion: batch/v1
kind: Job
metadata:
name: job-sleep-single
namespace: slurm-bridge
# slurm-bridge annotations on parent object
annotations:
slurmjob.slinky.slurm.net/job-name: job-sleep-single
spec:
completions: 1
parallelism: 1
template:
spec:
schedulerName: slurm-bridge-scheduler
containers:
- name: sleep
image: busybox:stable
command: [sh, -c, sleep 3]
resources:
requests:
cpu: '1'
memory: 100Mi
limits:
cpu: '1'
memory: 100Mi
restartPolicy: Never
schedulerName: slurm-bridge-scheduler is what routes the pod to
slurm-bridge. In a namespace listed in the chart’s
admission.managedNamespaces, which defaults to just slurm-bridge, the
admission controller replaces default-scheduler for you, so the field is
optional there. Explicitly selected custom schedulers are preserved. The example
sets the bridge scheduler explicitly so it also works in an unmanaged namespace.
Use a workload namespace separate from kube-system and the bridge’s
installation namespace, which the Helm webhooks exclude.
Every slurm-bridge annotation is prefixed with slurmjob.slinky.slurm.net/ and
must be set on the parent object. An annotation with any other prefix is
silently ignored, so a typo in the prefix costs you the setting with no error to
explain it. Any other Slurm job parameter is passed the same way, including the
time limit, account, partition, QoS and GRES. See
Annotations for the supported keys.
Let’s run this job:
❯ kubectl apply -f hack/examples/job/single.yaml
job.batch/job-sleep-single created
At this point, Kubernetes has dispatched our job, it was scheduled by Slurm, and executed to completion. Let’s take a look at each place that our job shows up.
On the Slurm side, we can observe the external job that was used to schedule our workload.
First, look at the job STATUS in Kubernetes:
$ kubectl get jobs -n slurm-bridge
NAME STATUS COMPLETIONS DURATION AGE
job-sleep-single Complete 1/1 8s 8m
Next, describe the job. Under the Events section, note the name of the pod on
which the job executed. Describe that pod:
$ kubectl describe job -n slurm-bridge job-sleep-single
Name: job-sleep-single
Namespace: slurm-bridge
Selector: batch.kubernetes.io/controller-uid=7cf47949-0099-4c1a-ab7e-d6e288283c82
Labels: batch.kubernetes.io/controller-uid=7cf47949-0099-4c1a-ab7e-d6e288283c82
batch.kubernetes.io/job-name=job-sleep-single
controller-uid=7cf47949-0099-4c1a-ab7e-d6e288283c82
job-name=job-sleep-single
Annotations: slurmjob.slinky.slurm.net/job-name: job-sleep-single
Parallelism: 1
Completions: 1
Completion Mode: NonIndexed
Start Time: Mon, 15 Sep 2025 11:38:48 -0600
Completed At: Mon, 15 Sep 2025 11:38:58 -0600
Duration: 10s
Pods Statuses: 0 Active (0 Ready) / 1 Succeeded / 0 Failed
Pod Template:
Labels: batch.kubernetes.io/controller-uid=7cf47949-0099-4c1a-ab7e-d6e288283c82
batch.kubernetes.io/job-name=job-sleep-single
controller-uid=7cf47949-0099-4c1a-ab7e-d6e288283c82
job-name=job-sleep-single
Containers:
sleep:
Image: busybox:stable
Port: <none>
Host Port: <none>
Command:
sh
-c
sleep 3
Limits:
cpu: 1
memory: 100Mi
Requests:
cpu: 1
memory: 100Mi
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 28m job-controller Created pod: job-sleep-single-w4dfl
Normal Completed 27m job-controller Job completed
Use the kubectl get pod command to get the labels for the pod in which the job
executed:
$ kubectl get pod -n slurm-bridge --show-labels
NAME READY STATUS RESTARTS AGE LABELS
job-sleep-single-w4dfl 0/1 Completed 0 31m batch.kubernetes.io/controller-uid=7cf47949-0099-4c1a-ab7e-d6e288283c82,batch.kubernetes.io/job-name=job-sleep-single,controller-uid=7cf47949-0099-4c1a-ab7e-d6e288283c82,job-name=job-sleep-single,scheduler.slinky.slurm.net/slurm-jobid=1
The scheduler.slinky.slurm.net/slurm-jobid label tells us that the Slurm JobID
for our job was 1:
scheduler.slinky.slurm.net/slurm-jobid=1
slurm@slurm-controller-0:/tmp$ scontrol show job 1
JobId=1 JobName=job-sleep-single
UserId=slurm(401) GroupId=slurm(401) MCS_label=kubernetes
Priority=1 Nice=0 Account=(null) QOS=normal
JobState=CANCELLED Reason=None Dependency=(null)
Requeue=1 Restarts=0 BatchFlag=0 Reboot=0 ExitCode=0:0
RunTime=00:00:08 TimeLimit=UNLIMITED TimeMin=N/A
SubmitTime=2025-07-10T15:52:53 EligibleTime=2025-07-10T15:52:53
AccrueTime=2025-07-10T15:52:53
StartTime=2025-07-10T15:52:53 EndTime=2025-07-10T15:53:01 Deadline=N/A
SuspendTime=None SecsPreSuspend=0 LastSchedEval=2025-07-10T15:52:53 Scheduler=Main
Partition=slurm-bridge AllocNode:Sid=10.244.5.5:1
ReqNodeList=(null) ExcNodeList=(null)
NodeList=slurm-bridge-1
BatchHost=slurm-bridge-1
StepMgrEnabled=Yes
NumNodes=1 NumCPUs=4 NumTasks=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:*
ReqTRES=cpu=1,mem=96046M,node=1,billing=1
AllocTRES=cpu=4,mem=96046M,node=1,billing=4
Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
Features=(null) DelayBoot=00:00:00
OverSubscribe=NO Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
Command=(null)
WorkDir=/tmp
AdminComment={"pods":"slurm-bridge/job-sleep-single-8wtc2/4035b798-e371-4b45-8d84-c9026568c7ec"}
OOMKillStep=0
The Command field is (null) because kubelet runs the workload instead of
Slurm. The external job remains running and holds its allocation while the Pod
runs. In this completed example, JobState=CANCELLED reflects the bridge’s
cleanup after the Pod terminated. Cancelling the Slurm job while its Pod is
still running causes the bridge to delete that Pod.
We can also look at this job using kubectl:
❯ kubectl describe job --namespace=slurm-bridge job-sleep-single
Name: job-sleep-single
Namespace: slurm-bridge
Selector: batch.kubernetes.io/controller-uid=8a03f5f6-f0c0-4216-ac0b-8c9b70c92eec
Labels: batch.kubernetes.io/controller-uid=8a03f5f6-f0c0-4216-ac0b-8c9b70c92eec
batch.kubernetes.io/job-name=job-sleep-single
controller-uid=8a03f5f6-f0c0-4216-ac0b-8c9b70c92eec
job-name=job-sleep-single
Annotations: slurmjob.slinky.slurm.net/job-name: job-sleep-single
Parallelism: 1
Completions: 1
Completion Mode: NonIndexed
Start Time: Thu, 10 Jul 2025 09:52:53 -0600
Completed At: Thu, 10 Jul 2025 09:53:02 -0600
Duration: 9s
Pods Statuses: 0 Active (0 Ready) / 1 Succeeded / 0 Failed
Pod Template:
Labels: batch.kubernetes.io/controller-uid=8a03f5f6-f0c0-4216-ac0b-8c9b70c92eec
batch.kubernetes.io/job-name=job-sleep-single
controller-uid=8a03f5f6-f0c0-4216-ac0b-8c9b70c92eec
job-name=job-sleep-single
Containers:
sleep:
Image: busybox:stable
Port: <none>
Host Port: <none>
Command:
sh
-c
sleep 3
Limits:
cpu: 1
memory: 100Mi
Requests:
cpu: 1
memory: 100Mi
Environment: <none>
Mounts: <none>
Volumes: <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 14m job-controller Created pod: job-sleep-single-8wtc2
Normal Completed 14m job-controller Job completed
As Kubernetes is the context in which this job actually executed, this is generally the more useful of the two outputs.
At this point, you should have a functional slurm-bridge cluster, and are
running jobs. Recommended next steps involve reviewing our documentation on
workloads