Config
Table of Contents
Overview
When using slurm-bridge, there are some configuration requirements and considerations to be made.
Partitions
Slurm bridge external jobs are submitted to a default partition (e.g.
slurm-bridge) in Slurm, or the one specified by
slurmjob.slinky.slurm.net/partition on the workload. Any partition where these
external jobs are submitted to should only consist of Slurm nodes that map to
Kubernetes nodes, whether they are nodes with co-located kubelet and slurmd or
Kubernetes nodes modeled as Slurm external nodes.
Nodes
Slurm bridge supports two ways to make Kubernetes nodes available to Slurm:
External nodes:
slurm-bridgeregisters labeled Kubernetes nodes with Slurm as external Slurm nodes. No slurm-operatorNodesetis required for the bridge partition.Hybrid nodes:
slurmdruns side-by-side with kubelet and registers a Slurm dynamic node. slurm-operator can manageslurmdwith a DaemonSet-modeNodeset, or host automation can install it as a native systemd service.
In both modes, Slurm should only schedule bridge jobs to Slurm nodes that map back to Kubernetes nodes.
External Nodes
External nodes are Kubernetes nodes that slurm-bridge registers with Slurm
using State=External. The node controller watches for the
scheduler.slinky.slurm.net/external-node label and creates or removes the
matching Slurm node.
apiVersion: v1
kind: Node
metadata:
name: worker-1
labels:
scheduler.slinky.slurm.net/external-node: "true"
annotations:
scheduler.slinky.slurm.net/external-node-partitions: slurm-bridge
The scheduler.slinky.slurm.net/external-node-partitions annotation is a
comma-separated list of Slurm partitions. The bridge validates that each
partition exists, then registers the Slurm node with matching features. Removing
a partition from the annotation removes its feature. Slurm Nodeset
configuration can then map those features into partitions:
Nodeset=slurm-bridge Feature=slurm-bridge
PartitionName=slurm-bridge Nodes=slurm-bridge State=UP Default=NO
Hybrid Nodes
Hybrid nodes run kubelet and slurmd on the same physical host. Bridge jobs run
on the dynamic Slurm nodes registered by those daemons. slurmd can run in a
slurm-operator pod or as a native host service. See
Host-installed Hybrid Slurm Nodes for a bare-metal
package and configless-registration procedure.
Hybrid nodes share capacity between workload managers over time. A physical node
must not run a native Slurm user workload and a Slurm-bridge-managed Kubernetes
user workload simultaneously. System components such as kubelet, slurmd, CNI,
device plugins, and monitoring DaemonSets are expected exceptions.
For Kubernetes-managed daemons, a DaemonSet-mode Nodeset can place one
slurmd pod on each worker node selected for bridge scheduling:
nodesets:
slurm-bridge:
enabled: true
scalingMode: DaemonSet
partition:
enabled: true
podSpec:
nodeSelector:
scheduler.slinky.slurm.net/slurm-bridge: worker
For DRA-backed devices, the bridge node controller records the stable device
inventory in the Slurm node’s Extra field. The hybrid node’s gres.conf must
provide matching Name, Type, and Count values. Other GRES entries are
allowed and remain available to native Slurm workloads. If the DRA-managed
entries do not match, the controller sets the Kubernetes node’s
SlinkySlurmGRESCompatible condition to False with the required gres.conf
inventory. It does not check device order or Cores=, which
NUMA alignment also needs. This condition reports
compatibility to administrators; Slurm node features control bridge-job
placement.
The controller adds the slurm_bridge_gres_compatible Slurm feature to every
external node and to hybrid nodes whose GRES configuration has been verified.
Every Slurm job submitted by the bridge requires this feature. An incompatible
hybrid node therefore remains available to native Slurm jobs while bridge jobs
cannot be allocated to it. Other hybrid-node features and native job constraints
are unaffected.
Topology
Slurm supports dynamic node topology with topology.yaml. The topology file
must be available to Slurm. Kubernetes nodes can be annotated with the Slurm
topology units they belong to.
configFiles:
topology.yaml: |
---
- topology: topo-switch
cluster_default: true
tree:
switches:
- switch: sw_root
children: s[1-2]
- switch: s1
nodes: slurm-node[1-2]
- switch: s2
nodes: slurm-node[3-4]
- topology: topo-block
cluster_default: false
block:
block_sizes:
- 2
- 4
blocks:
- block: b1
nodes: slurm-node[1-2]
- block: b2
nodes: slurm-node[3-4]
Annotate the Kubernetes node with the topology units for the corresponding Slurm node:
apiVersion: v1
kind: Node
metadata:
name: worker-1
annotations:
topology.slinky.slurm.net/spec: topo-switch:s1,topo-block:b1
External nodes and hybrid nodes managed by slurm-operator use the same Kubernetes annotation, but different controllers apply it to Slurm:
In external mode,
slurm-bridgereadstopology.slinky.slurm.net/specfrom the labeled Kubernetes node and reconciles the external Slurm node’s dynamic topology.For hybrid nodes managed by slurm-operator, the operator reads
topology.slinky.slurm.net/specfrom the Kubernetes node where theNodesetpod is scheduled, copies it to the pod, and reconciles theslurmdpod’s Slurm node topology.
Removing the annotation clears the dynamic topology for those nodes.
Host-installed hybrid daemons require administrators to configure their Slurm
node topology directly; neither controller applies the annotation to them. When
using multiple topologies, Slurm partitions can select a specific topology with
the partition Topology setting; otherwise Slurm uses the cluster_default
topology.
Hybrid Workload Isolation
Bridge jobs receive exclusive whole-node Slurm allocations by default. Workloads
requesting slurmjob.slinky.slurm.net/exclusive: "false" are always submitted
with Shared=mcs and the configured schedulerConfig.mcsLabel. This allows
bridge-managed Kubernetes workloads in the same MCS category to share a node
without enabling unprotected sharing with native Slurm jobs.
Configure both the bridge and Slurm to enable MCS isolation:
# slurm-bridge values.yaml
schedulerConfig:
mcsLabel: kubernetes
# slurm.conf
MCSPlugin=mcs/label
MCSParameters=ondemand,ondemandselect
With ondemandselect, the bridge’s Shared=mcs value activates MCS node
filtering for non-exclusive jobs. Native Slurm jobs with a different or empty
MCS label cannot share those nodes while a kubernetes-labeled allocation is
running.