Workloads
Table of Contents
Overview
In Slurm, all workloads are represented by jobs. In slurm-bridge, however,
there are a number of forms that workloads can take. While workloads can still
be submitted as a Slurm job, slurm-bridge also enables users to submit
workloads through Kubernetes. Most workloads that can be submitted to
slurm-bridge from within Kubernetes are represented by an existing Kubernetes
batch workload primitive.
At this time, slurm-bridge has scheduling support for Jobs,
JobSets, Pods, built-in PodGroup
(scheduling.k8s.io/v1alpha2 on Kubernetes 1.36, or
scheduling.k8s.io/v1beta1 on 1.37),
PodGroup coscheduling (scheduler-plugins,
deprecated), and LeaderWorkerSets. If your workload requires or
benefits from co-scheduled pod launch (e.g. MPI, multi-node), use
built-in PodGroup on Kubernetes 1.36 or 1.37 when its API is
enabled. The deprecated PodGroup coscheduling
integration remains available for compatibility when the built-in API is
unavailable.
Using the slurm-bridge Scheduler
slurm-bridge uses an
admission controller
to control which resources are scheduled using the slurm-bridge-scheduler. The
slurm-bridge-scheduler is designed as a non-primary scheduler and is not
intended to replace the default
kube-scheduler.
The slurm-bridge-scheduler only schedules pods whose spec.schedulerName
matches its configured name, slurm-bridge-scheduler by default. In managed
namespaces, admission changes default-scheduler to that name; explicitly
selected custom schedulers are preserved.
Alternatively, a pod can specify Pod.Spec.schedulerName=slurm-bridge-scheduler
from an unmanaged namespace to select the slurm-bridge-scheduler. The Helm
webhooks exclude kube-system and the bridge’s installation namespace, so use a
separate namespace for bridge workloads.
Please review slurm-bridge admission controller to learn
more.
Device resources
Supported DRA DeviceClasses
slurm-bridge supports the following DRA DeviceClass extended resources out of
the box:
DeviceClass |
Extended resource |
Device type |
|---|---|---|
|
|
CPU |
|
|
NVIDIA GPU |
For these resources, slurm-bridge translates the Slurm allocation into a DRA
ResourceClaim and records the allocated devices for the Pod. Additional indexed
DeviceClasses are supported when they resolve to a configured device profile.
Other DeviceClass extended resources are unsupported. Validation covers requests
and limits in both init containers and regular containers. For NVIDIA GPUs and
MIG devices, see NVIDIA GPUs and MIG with DRA.
The bridge keeps DRADeviceTaintRules disabled by default because clusters
older than Kubernetes 1.37 do not serve resource.k8s.io/v1 DeviceTaintRules.
Slurm chooses the devices, so Kubernetes device taints do not stop Slurm from
allocating a tainted device.
Configuration from each matching DeviceClass.spec.config is copied into the
generated ResourceClaim allocation with source FromClass and scoped to the
requests which use that class. Claim configuration is retained with source
FromClaim after the class configuration. This preserves opaque driver
configuration such as network attachment parameters while keeping allocation
decisions under Slurm’s control.
Indexed DRA devices are mapped to Slurm GRES through deviceProfiles in the
shared Slurm Bridge configuration. The built-in profiles cover CPU, NVIDIA GPU,
and PCI-backed DRANET devices. The example GPU driver requires explicit
configuration. Setting sharedConfig.deviceProfiles replaces the built-in list;
include the built-in entries too if they are still needed. For example, this
configuration enables only the custom accelerator profile:
sharedConfig:
deviceProfiles:
- name: custom-accelerator
driver: accelerator.example.com
selector: device.driver == 'accelerator.example.com'
backend:
type: indexed-gres
gresName: accelerator
The DeviceClass must have exactly one CEL selector and it must exactly match a
configured profile selector. Profile names become Slurm GRES types: they must
start and end with an alphanumeric character, may contain only alphanumeric
characters, ., _, and -, and must remain stable while allocations using
them exist. driver must be a valid lowercase Kubernetes DRA driver name, and
gresName must be a DNS-1123 label of at most 60 characters because it is also
used as the ResourceClaim request name. Every configured backend.gresName must
also be listed in Slurm’s GresTypes; for example, profiles using gpu and
nic require GresTypes=gpu,nic. Slurm omits a GRES from its node inventory
when its type is not listed. The bundled external and hybrid Kind configurations
include GresTypes=gpu,nic by default. Selectors use the same length and
estimated-cost limits as Kubernetes DeviceClass CEL selectors. Profiles for the
same driver must be mutually exclusive. If a device matches more than one
profile, the node controller leaves its Slurm GRES inventory unchanged and emits
an OverlappingDRADeviceProfiles Warning event on the Kubernetes Node.
The SR-IOV example shows how to add your own DeviceProfile, configure a matching DeviceClass, and request its devices in a workload. Adapt the profile and network configuration to your driver and cluster; the bundled network setup is for Kind.
The built-in NVIDIA profile uses gpu.nvidia.com as its name and maps to
gpu:gpu.nvidia.com in Slurm. The explicitly configured example GPU profile
uses gpu.example.com and maps to gpu:gpu.example.com. These preserve the
driver-named GRES types used before DeviceProfiles. Registered profiles require
the applied device-index mapping in the Slurm node’s Extra field before
allocations can bind.
ResourceSlices must explicitly name a single node with a nonempty
spec.nodeName. spec.nodeSelector, spec.allNodes, and
spec.perDeviceNodeSelection are unsupported. All slices in the latest
generation of a driver/pool must name the same node and agree on
resourceSliceCount. That generation must be complete before the bridge uses
its devices. Multiple slices per pool are supported, including CPU driver pools;
an incomplete new generation does not fall back to an older generation.
The default dranet-rdma profile maps PCI-backed devices with DRANET’s rdma
attribute set to the Slurm nic GRES. It includes InfiniBand, RoCE, and iWARP
devices; it does not imply an InfiniBand link layer.
Enable DRANET’s PersistentResourceSliceAttributes feature gate to retain
allocated devices and their attributes in ResourceSlices. It is alpha and
disabled by default in DRANET v1.5.0; hack/kind.sh --dranet enables it
automatically. For the DRANET v1.5.0 Helm chart, set:
args:
featureGates: PersistentResourceSliceAttributes=true
Append any other enabled DRANET gates to the same comma-separated value. Without
persistence, moving a NIC into a Pod can remove the device or its
profile-matching attributes from ResourceSlices. The bridge then drains the
external Slurm node for re-registration: scontrol show node <node> shows
DRAIN and a reason containing removing external node, while unrelated jobs
remain pending despite free CPUs. If this coincides with NIC allocation and
clears on release, check the DRANET feature gate and ResourceSlice inventory.
The Kind e2e device profiles explicitly add
the example GPU driver and a fixture-only dranet0 profile alongside the
built-in profiles. The dranet0 profile selects the dummy network interface
created by hack/kind.sh --dranet by driver and interface name:
sharedConfig:
deviceProfiles:
- name: dranet0
driver: dra.net
selector: >-
device.driver == 'dra.net' && has(device.attributes['dra.net'].ifName) &&
device.attributes['dra.net'].ifName == 'dranet0'
backend:
type: indexed-gres
gresName: nic
Legacy GPU device plugins
The following non-DRA extended resources are also supported:
Extended resource |
Device plugin |
|---|---|
|
NVIDIA GPU |
|
AMD GPU |
These resources use the generic device-plugin path unless a DeviceClass alias
resolves to a configured bridge DeviceProfile. Aliases without a matching
profile retain their device-plugin meaning. On the device-plugin path,
quantities are translated to a generic Slurm gres/gpu=<count> request. Slurm
selects and reserves the node and GPU count, while kubelet and the device plugin
choose the physical devices. No DRA ResourceClaim is created, so Slurm and
Kubernetes do not coordinate exact GPU identities on this path.
CPU DRA
Native cpu requests do not activate the CPU DRA driver. To request CPUs from
the dra.cpu DeviceClass, specify its extended resource explicitly:
resources:
requests:
deviceclass.resource.kubernetes.io/dra.cpu: "2"
limits:
deviceclass.resource.kubernetes.io/dra.cpu: "2"
The extended resource quantity is used as the Slurm CPU count. A Pod that
requests this resource cannot also specify native cpu requests or limits.
Set slurmjob.slinky.slurm.net/exclusive: "false" on the workload’s
annotation source. Exclusive allocation is the default and is
unsupported for CPU DRA because the driver needs unallocated CPUs for shared
containers. Non-exclusive claims can also exhaust that pool; the driver rejects
allocations that would leave shared containers without CPUs.
CPU DRA constrains the container to Slurm’s allocated CPU set. The CPU driver also removes DRA-allocated CPUs from the shared CPU sets of running native containers. Native CPU requests still reserve capacity in Slurm, but native containers share all CPUs not claimed through DRA; Slurm’s native CPU IDs do not define their container CPU sets.
CPU DRA requires Kubernetes 1.36 or newer with DRAExtendedResource and
DRAConsumableCapacity enabled, plus containerd with CDI and NRI enabled. Use
the pinned CPU driver with
cpuDeviceMode: grouped, groupBy: numanode or groupBy: socket, and
allocator: external. Use numanode for NUMA alignment.
After reserved CPUs are excluded, groups (NUMA domains or physical sockets) must
have equal core counts and cores must have equal thread counts. Slurm must be
the sole allocator for these devices; kubelet’s CPU Manager must not assign the
same CPUs independently.
To migrate from individual CPU devices, drain Slurm nodes and wait for all claims to be unprepared before replacing the driver and bridge. Use the same procedure before changing CPU reservations or topology. Resume work after Slurm registration matches the new inventory. Hybrid nodes also require matching slurmd geometry; geometry checks do not verify Linux CPU ordering on physical hardware.
NUMA alignment
slurm-bridge binds each DRA device to the Slurm cores of its NUMA node, so
Slurm can place a job’s CPUs, GPUs, and NICs on the same NUMA node. This
requires:
CPU DRA with
groupBy: numanode.Kubernetes 1.37 or newer with the
DRAListTypeAttributesfeature gate on the API server, scheduler, controller manager, and kubelet.DRA drivers which publish
resource.kubernetes.io/numaNode, such as the NVIDIA GPU driver and DRANET with theirDRAListTypeAttributesfeature gate.
Alignment is best effort: Slurm prefers local cores, but a job which needs more than one NUMA node still runs. With NUMA-grouped CPU inventory, each NUMA domain is registered as a Slurm socket. This preserves device locality when several NUMA domains share a physical socket, including sub-NUMA clustering. Abstract Slurm cores are ordered by physical socket, NUMA domain, and then core; each domain occupies a contiguous range.
When changing from physical-socket registration to NUMA domains as sockets, drain nodes and wait for claims to be unprepared. External nodes are re-registered with the new geometry before accepting bridge jobs.
Hybrid nodes are aligned only when the site’s gres.conf describes the same
topology; slurm-bridge does not write it, and the SlinkySlurmGRESCompatible
condition checks only GRES names, types, and counts. For each GRES type, list
devices in the string order of their DRA device names (gpu-10 before gpu-2),
give each the Cores= of its NUMA domain, and set slurmd’s Sockets and
CoresPerSocket to match CPU DRA. For sub-NUMA clustering, configure
SlurmdParameters=numa_node_as_socket and update the node geometry to match
slurmd -C --parameters=numa_node_as_socket. Geometry mismatches prevent bridge
jobs from using the node. Verify Linux CPU ordering against the bridge mapping
as well; geometry checks alone cannot establish it. Incorrect GRES locality can
still let jobs run with devices and CPUs from different NUMA domains.
Annotations
Users can influence how slurm-bridge represents their Kubernetes workload in
Slurm by adding slurmjob.slinky.slurm.net/* annotations to Pods, including
through workload Pod templates. They configure the Slurm external job and are
separate from scheduler-managed metadata.
Example “pause” bare pod to illustrate annotations:
apiVersion: v1
kind: Pod
metadata:
name: pause
# Slurm job annotations on this Pod
annotations:
slurmjob.slinky.slurm.net/timelimit: "5"
slurmjob.slinky.slurm.net/account: foo
spec:
schedulerName: slurm-bridge-scheduler
containers:
- name: pause
image: registry.k8s.io/pause:3.6
resources:
limits:
cpu: "1"
memory: 100Mi
Example “sleep” Job to illustrate annotations:
apiVersion: batch/v1
kind: Job
metadata:
name: sleep
spec:
template:
metadata:
annotations:
slurmjob.slinky.slurm.net/timelimit: "5"
slurmjob.slinky.slurm.net/account: foo
spec:
schedulerName: slurm-bridge-scheduler
restartPolicy: Never
containers:
- name: sleep
image: busybox:stable
command: [sh, -c, sleep 3]
resources:
limits:
cpu: "1"
memory: 100Mi
Resolution rules
Slurm parameters are resolved from lowest to highest precedence: scheduler or Slurm defaults -> values derived from the workload and Pods -> legacy workload annotations -> Pod annotations. Pod annotations always win. Values are merged before parsing, so an overridden legacy value is ignored even if it is invalid.
For built-in PodGroups, legacy annotation precedence remains selected workload owner -> PodGroup -> Workload, followed by Pod annotations. Keys that do not conflict are combined. The first Pod in each Slurm component supplies the Pod annotations. Pods within a component must have matching Slurm annotations; consistency is not checked. For CompositePodGroup gangs, each leaf PodGroup has its own component and can use different annotations from sibling PodGroups.
Top-level workload annotations are being deprecated and remain defaults during the transition; support will be removed in a future release. Admission warns about them when known workloads are created in managed namespaces by default; it can also reject them. Move them to Pod template metadata; for LeaderWorkerSets, use the leader and worker templates as appropriate.
Supported Slurm job annotations
Purpose |
Annotation suffixes |
|---|---|
Identity and accounting |
|
Scheduling policy |
|
Resources |
|
Naming and duration |
|
Prefix every suffix with slurmjob.slinky.slurm.net/; for example,
slurmjob.slinky.slurm.net/account. Node counts and priority are base-10
integers. CPU and memory accept Kubernetes quantities. gres is passed to
Slurm’s tres_per_node field; use a value such as gres/gpu=2. timelimit
accepts a duration with a unit suffix (s, m, h, d, w), one of Slurm’s
--time formats (MM:SS, HH:MM:SS, D-HH, D-HH:MM, D-HH:MM:SS), or a
bare integer, which is read as minutes; sub-minute values are rounded up to one
minute because Slurm’s time limit has minute granularity. Exclusive placement is
the default. exclusive: "false" always requests MCS-category sharing; the
resulting external job uses Slurm’s Shared=mcs mode and requires a configured
schedulerConfig.mcsLabel. Because Slurm’s mcs/label plugin does not
authorize label use, production clusters must also reserve that label from
native users as described in
Production label authorization.
Every bridge job requires the slurm_bridge_gres_compatible node feature, so
the scheduler combines it with the constraints value using Slurm’s AND
operator. Slurm’s constraint grammar does not allow nested parentheses, allows
one set of matching-OR brackets, and only accepts a feature count outside
brackets when it is the sole feature. Within those rules any expression is
supported, including a&b, a|b, a&(b|c), [rack1|rack2] and
[a100*2&h100*1]. Admission rejects the few expressions that cannot be combined
with an additional feature: a bare count such as rack1*2, and an OR outside
parentheses mixed with parentheses or brackets such as (a&b)|(c&d). Rewrite
these with a single level of grouping, for example [a&b|c&d].
The user-id, group-id, account, and exclusive annotations apply only at
submission. Changing them does not modify an existing Slurm job; submit a new
workload to use different values. Bridge preserves the submitted ownership and
isolation when updating the job through its privileged Slurm identity.
Other annotations can update a Slurm job while it is pending. Slurm validates each change, but a failed update can still have partially applied changes. Removing an annotation does not necessarily reset the stored Slurm value. Once Slurm allocates the job, treat its annotations as fixed. Pod membership is fixed at submission, as described in Changing the number of Pods.
Scheduler-managed Pod metadata
Slurm-bridge writes the following metadata to Pods as it creates and allocates the external job:
Metadata |
Type |
Meaning |
|---|---|---|
|
Label |
Slurm job ID associated with the Pod. All Pods represented by one external job receive the same value. |
|
Annotation |
Kubernetes node selected from the Slurm allocation for this Pod. |
These keys are owned by slurm-bridge and must not be edited manually. Admission rejects non-empty values when creating a managed Pod. Subsequent updates are not rejected on that basis; the Pod controller repairs metadata using the verified Slurm allocation.
Pod grouping
Slurm-bridge turns each group of Kubernetes Pods into one Slurm external job:
Workload |
Pods in one external job |
Where to put annotations |
|---|---|---|
Pod |
That Pod |
Pod metadata |
Job or JobSet |
One Pod |
Pod template metadata |
Built-in PodGroup |
Pods with the same |
Member Pod metadata |
PodGroup coscheduling |
Pods with the same |
Member Pod metadata |
LeaderWorkerSet |
One LeaderWorkerSet group |
Pod template metadata |
Other controller |
One Pod |
Pod template metadata |
The grouping precedence is built-in PodGroup -> PodGroup coscheduling -> supported workload owner -> the Pod itself. Native PodGroups use their explicit Workload reference and CompositePodGroup parent references.
Legacy owner lookups follow only Pod -> Job -> JobSet and the defined LeaderWorkerSet paths, directly or through StatefulSets and leader Pods. The selected Job, JobSet, or LeaderWorkerSet supplies defaults; intermediate objects do not. Native PodGroups also read their explicit PodGroup and Workload defaults. Coscheduling PodGroups retain their PodGroup defaults.
Other controller owners
Other controller references are not fetched or traversed. Deployment/ReplicaSet, RayCluster, and custom controller chains use Pod annotations and schedule one external job per Pod. For Ray, place annotations on the head and worker Pod templates; RayCluster metadata is not read. Unknown owners above supported workloads are ignored. Errors reading objects on a defined path still fail scheduling.
Workload guides
Built-in PodGroup: Workload and PodGroup APIs, gang scheduling, and Kubernetes version upgrades.
JobSets: per-Pod allocations, scaling restrictions, and recovery.
LeaderWorkerSet: group scheduling, replica scaling, and recovery.
NVIDIA GPUs and MIG with DRA: driver installation, MIG device profiles, and allocation checks.
TPU DRA: per-Pod whole-host requests and a fake-device Kind check.
PodGroup coscheduling: legacy scheduler-plugins integration and migration to built-in PodGroups.
Workload Preemption
PreemptMode=REQUEUE and PreemptMode=CANCEL currently have the same effect on
slurm-bridge workloads. Slurm terminates the external job rather than
requeueing it, and the bridge deletes its non-terminal, bound Pods. Pod
termination is asynchronous.
Workload controllers determine whether Pods are replaced:
Standalone Pods and PodGroup or CompositePodGroup members without an owning workload controller are not recreated.
Jobs and JobSets recover according to Job retry budgets and JobSet failure policies. Preempted Pods count as failed attempts, so
backoffLimit: 0can cause the workload to fail.LeaderWorkerSets recreate groups according to their group recovery policy.
Replacement Pods need scheduling again and request new Slurm allocations. Slurm-bridge does not resume the preempted Pods or requeue their original allocations.
Scaling and failure behavior
Workload controllers or external autoscalers create and remove Pods; the bridge submits, updates, and cancels their Slurm external jobs. An accepted Kubernetes update does not guarantee that Slurm can allocate the requested resources.
A JobSet normally has one Slurm job per Pod, while direct LeaderWorkerSet scheduling has one per replica group. A Gang PodGroup takes precedence over those defaults. A Basic PodGroup retains the normal owner-based allocation boundary. The behavior below applies to each allocation independently.
CPU and memory changes
In-place changes to an existing Pod’s CPU or memory requests/limits are
unsupported both before and after Slurm allocation. The admission webhook
rejects updates through pods/resize with
can't resize a Slurm Bridge-managed pod. A Pod still in Kubernetes Pending
phase may already have a Slurm allocation, so its phase does not indicate that
resizing is allowed.
Use replacement Pods with the desired resource template. Replacement Pods or groups request new Slurm allocations; the bridge does not resize the old allocations. This also applies when the old Pods have not started running yet. See the JobSet and LeaderWorkerSet guides for workload-specific replacement behavior.
Changing the number of Pods
Change |
Before Slurm allocates the job |
After Slurm allocates the job |
|---|---|---|
Change membership of an existing gang |
Membership is fixed when the Slurm job is submitted. Later Pods cannot join the pending job. |
Live expansion or shrinking is unsupported. Recreate the group for a new allocation. |
Add or remove independent Pods, including ungrouped JobSet Pods when the API permits it |
Each added Pod submits its own job; removed Pods have their pending jobs canceled during cleanup. |
New Pods queue for separate allocations; removed Pods release their own allocations during cleanup. Other allocations continue. |
Gang quorum is spec.schedulingPolicy.gang.minCount for built-in PodGroups,
spec.minMember for legacy PodGroups, or the configured LWS group size for
implicit LWS gangs. A CompositePodGroup also applies
spec.schedulingPolicy.gang.minGroupCount at each level of its hierarchy.
Quorum is a readiness threshold, not a cap: submission records the Pod UIDs and
child groups present in the submitted batch.
Later Pods and child groups wait for the current allocation to finish and a new
batch to meet quorum. They cannot join or block its original members. The
controller reports WaitingForNextAllocation; a long-running allocation can
leave late arrivals waiting indefinitely. Concurrent batches under one gang
identity are unsupported. Set quorum to the intended total when all members must
share an allocation.
For LWS, scale whole replica groups. Replica groups combined into one Gang PodGroup share an allocation and cannot scale independently.
Scale-down and resource release
Deleting Pods does not partially shrink a running Slurm allocation. The
registration controller cancels the Slurm job once all Pods associated with that
job are terminal (Succeeded or Failed) or are being deleted before binding
to a node. Bound Pods must terminate before their allocation is released.
Periodic cleanup also cancels pending or running jobs whose referenced Pods no
longer exist. Removing one independent Pod therefore releases its allocation,
while removing a whole LWS group releases that group’s allocation. Removing only
some Pods from a shared job retains its full allocation while other members
remain active.
Cleanup is asynchronous: a deletion request or a Pod in Terminating does not
mean Slurm resources are already free. The bridge also removes its Pod finalizer
and generated DRA ResourceClaim during terminal Pod cleanup.
Failures and recovery
If a scale-up cannot obtain resources, new Pods and their jobs wait for scheduling; existing independent allocations continue. Admission rejection leaves the Kubernetes object unchanged. If Slurm rejects a pending job update, the previous Slurm request remains in effect. Submission errors are reported by the registration controller through workload events and controller logs; update errors appear in scheduling failures and scheduler logs. Both can be retried. The bridge does not roll back the workload’s desired replica count or templates, and an API error is not evidence that a job was canceled or resources were released.
A container restart within the same Pod does not by itself release the Slurm job. Terminal Pods trigger the cleanup described above. If a Slurm job stops running, for example because it is canceled or reaches its time limit, the registration controller deletes its non-terminal, bound Pods. Workload controllers determine whether Pods are replaced; see JobSet recovery and LeaderWorkerSet recovery. Replacement Pods need scheduling again. Do not rely on individual Pod replacement to extend or repair an allocated gang; recreate the whole group for a fresh allocation.
When a Pod remains SchedulingGated, inspect its SlurmRegistrationPending
event for the group to check. Registration events such as WaitingForMembers
and SlurmRegistrationFailed are recorded on that group, including the parent
CompositePodGroup for composite workloads. Also inspect the controllers’ logs.
When scaling stalls, inspect Pod scheduling events, JobSet or LWS status, bridge
scheduler and controller logs, and the Slurm job in squeue or
scontrol show job. The Pod label scheduler.slinky.slurm.net/slurm-jobid
identifies its Slurm job. Correct rejected settings or recreate the affected
group, and verify that the old job is no longer pending or running before
assuming its resources have been released.