Config

Table of Contents

Overview

When using slurm-bridge, there are some configuration requirements and considerations to be made.

Partitions

Slurm bridge external jobs are submitted to a default partition (e.g. slurm-bridge) in Slurm, or the one specified by slurmjob.slinky.slurm.net/partition on the workload. Any partition where these external jobs are submitted to should only consist of Slurm nodes that map to Kubernetes nodes, whether they are nodes with co-located kubelet and slurmd or Kubernetes nodes modeled as Slurm external nodes.

Nodes

Slurm bridge supports two ways to make Kubernetes nodes available to Slurm:

  • External nodes: slurm-bridge registers labeled Kubernetes nodes with Slurm as external Slurm nodes. No slurm-operator Nodeset is required for the bridge partition.

  • Hybrid nodes: slurm-operator runs slurmd side-by-side with kubelet, typically using a DaemonSet-mode Nodeset. The slurmd pods register Slurm dynamic nodes.

In both modes, Slurm should only schedule bridge jobs to Slurm nodes that map back to Kubernetes nodes.

External Nodes

External nodes are Kubernetes nodes that slurm-bridge registers with Slurm using State=External. The node controller watches for the scheduler.slinky.slurm.net/external-node label and creates or removes the matching Slurm node.

apiVersion: v1
kind: Node
metadata:
  name: worker-1
  labels:
    scheduler.slinky.slurm.net/external-node: "true"
  annotations:
    scheduler.slinky.slurm.net/external-node-partitions: slurm-bridge

The scheduler.slinky.slurm.net/external-node-partitions annotation is a comma-separated list of Slurm partitions. The bridge validates that each partition exists, then registers the Slurm node with matching features. Slurm Nodeset configuration can then map those features into partitions:

Nodeset=slurm-bridge Feature=slurm-bridge
PartitionName=slurm-bridge Nodes=slurm-bridge State=UP Default=NO

Hybrid Nodes

Hybrid nodes run kubelet and slurmd on the same physical host. In this mode, slurm-operator manages the Slurm nodes with a Nodeset, and bridge jobs run on the Slurm nodes registered by those slurmd pods.

Hybrid nodes share capacity between workload managers over time. A physical node must not run a native Slurm user workload and a Slurm-bridge-managed Kubernetes user workload simultaneously. System components such as kubelet, slurmd, CNI, device plugins, and monitoring DaemonSets are expected exceptions.

For example, a DaemonSet-mode Nodeset can place one slurmd pod on each Kubernetes worker node selected for bridge scheduling:

nodesets:
  slurm-bridge:
    enabled: true
    scalingMode: DaemonSet
    partition:
      enabled: true
    podSpec:
      nodeSelector:
        scheduler.slinky.slurm.net/slurm-bridge: worker

Topology

Slurm supports dynamic node topology with topology.yaml. The topology file must be available to Slurm. Kubernetes nodes can be annotated with the Slurm topology units they belong to.

configFiles:
  topology.yaml: |
    ---
    - topology: topo-switch
      cluster_default: true
      tree:
        switches:
          - switch: sw_root
            children: s[1-2]
          - switch: s1
            nodes: slurm-node[1-2]
          - switch: s2
            nodes: slurm-node[3-4]
    - topology: topo-block
      cluster_default: false
      block:
        block_sizes:
          - 2
          - 4
        blocks:
          - block: b1
            nodes: slurm-node[1-2]
          - block: b2
            nodes: slurm-node[3-4]

Annotate the Kubernetes node with the topology units for the corresponding Slurm node:

apiVersion: v1
kind: Node
metadata:
  name: worker-1
  annotations:
    topology.slinky.slurm.net/spec: topo-switch:s1,topo-block:b1

External and hybrid modes use the same Kubernetes annotation, but different controllers apply it to Slurm:

  • In external mode, slurm-bridge reads topology.slinky.slurm.net/spec from the labeled Kubernetes node and reconciles the external Slurm node’s dynamic topology.

  • In hybrid mode, slurm-operator reads topology.slinky.slurm.net/spec from the Kubernetes node where the Nodeset pod is scheduled, copies it to the pod, and reconciles the slurmd pod’s Slurm node topology.

Removing the annotation clears the dynamic topology in either mode. When using multiple topologies, Slurm partitions can select a specific topology with the partition Topology setting; otherwise Slurm uses the cluster_default topology.

Hybrid Workload Isolation

Bridge jobs receive exclusive whole-node Slurm allocations by default. Workloads requesting slurmjob.slinky.slurm.net/exclusive: "false" are always submitted with Shared=mcs and the configured schedulerConfig.mcsLabel. This allows bridge-managed Kubernetes workloads in the same MCS category to share a node without enabling unprotected sharing with native Slurm jobs.

Configure both the bridge and Slurm to enable MCS isolation:

# slurm-bridge values.yaml
schedulerConfig:
  mcsLabel: kubernetes
# slurm.conf
MCSPlugin=mcs/label
MCSParameters=ondemand,ondemandselect

With ondemandselect, the bridge’s Shared=mcs value activates MCS node filtering for non-exclusive jobs. Native Slurm jobs with a different or empty MCS label cannot share those nodes while a kubernetes-labeled allocation is running.

Production label authorization

Slurm’s mcs/label plugin controls category-based sharing but accepts arbitrary labels; it does not authorize their use. Production clusters must reserve schedulerConfig.mcsLabel at the slurmctld boundary with a server-side job_submit plugin or equivalent site policy, using a trusted bridge identity and rejecting native submissions or modifications that request the reserved label. Without that Slurm-side policy, a native user can deliberately join the bridge’s MCS category.

MCS only governs workloads represented by active Slurm allocations. Continue to use the slinky.slurm.net/managed-node NoExecute taint and admission policy to keep Kubernetes workloads that bypass slurm-bridge off hybrid nodes. Operational or controller-driven cancellation must also keep a node unavailable to native Slurm work until its Kubernetes pods have actually stopped.