Controllers
Table of Contents
Overview
The scheduler process runs two controllers:
Node Controller - Registers and taints managed Kubernetes nodes.
Registration Controller - Creates Slurm allocations, publishes Pod registration metadata, and cleans up completed workloads.
Both are enabled by default.
When upgrading from separate deployments, move controllers resource,
placement, and registrationWorkers overrides to scheduler. The Helm
pre-upgrade hook stops the old components; if upgradeHook.enabled=false, stop
them manually before upgrading.
The chart always enables leader election. Standalone binaries use
--leader-elect; KubeSchedulerConfiguration.leaderElection.leaderElect and
the old scheduler.leaderElect Helm value are ignored. The lease uses
leaderElection.resourceNamespace from the scheduler configuration (default:
kube-system). Override it with --leader-election-namespace. Helm uses the
release namespace.
Node Controller
The node controller taints managed nodes and synchronizes their scheduling state with Slurm.
A managed node is a Kubernetes node with a corresponding Slurm node. This
includes both external and hybrid nodes; only hybrid nodes
require a colocated slurmd.
Node Registration and Partition Assignment
The node controller supports dynamic node registration in Slurm through Kubernetes node labels:
Node Registration Label (
scheduler.slinky.slurm.net/external-node): When this label is present on a Kubernetes node, the node controller will automatically register the node in Slurm with the correct CPU and memory resources. When the label is removed, the node will be removed from Slurm.Node Partitions Annotation (
scheduler.slinky.slurm.net/external-node-partitions): Specifies the Slurm partition(s) the node should be assigned to. The value is a comma-separated list of partition names. When this annotation is set, the node controller will:Validate that all specified partitions exist in Slurm
Register the node with node features matching the partition names
Allow Slurm’s NodeSet configuration in
slurm.confto assign the node to those partitions
Example:
apiVersion: v1 kind: Node metadata: name: worker-1 labels: scheduler.slinky.slurm.net/external-node: "true" annotations: scheduler.slinky.slurm.net/external-node-partitions: "slurm-bridge,gpu"
This will register the node with features
slurm-bridge,gpu, allowing it to be assigned to those partitions inslurm.confusing NodeSet features.For external nodes, the bridge owns the Slurm node’s features: they are the partitions above, plus the features in the
feature.slinky.slurm.net/specannotation, plus the bridge’s own compatibility feature. A feature removed from either annotation is removed from the Slurm node, and features set directly in Slurm are replaced. Hybrid nodes (those registered by their ownslurmd) keep the features and topology they registered with.Note: The controller will error if any specified partition does not exist in Slurm. Partitions must be pre-configured in Slurm’s configuration.
Registration Controller
The registration controller creates held Slurm allocations and removes Pod registration gates once allocation metadata is published. It also repairs metadata and cleans up completed allocations.
Cleanup retains recorded, published, or bound members. Before cancellation it
confirms membership through the Kubernetes API and ownership through Slurm. A
deleting, bound Pod retains its allocation until it becomes terminal. A Pod on a
node with no working kubelet never does; once its containers have stopped, taint
the node with node.kubernetes.io/out-of-service=nodeshutdown:NoExecute. Pod
garbage collection then marks the Pod Failed and cleanup releases its job.
Running heterogeneous jobs can release individual nonzero components. Component
zero and pending heterogeneous jobs are retained while any member needs the
allocation.
Bound Pods whose jobs stop running are deleted after a live Slurm check. Claims and scheduler finalizers are removed only after Slurm reports the allocation finished or absent; a cancellation acknowledgement alone does not release them.
Configure concurrency with scheduler.registrationWorkers in Helm or
--registration-workers on the combined scheduler binary. The default is 10
workers. The Node controller also defaults to 10 workers, configurable with
--node-workers. scheduler.podWorkers and --pod-workers remain accepted for
compatibility but have no effect.
Submission Failures
When Slurm refuses a submission, the Pods stay behind their registration gate. The controller records an Event with Slurm’s message on the Pod or its group object, such as a PodGroup, and retries later:
SlurmRegistrationRejected(Warning): Slurm refused the request itself, such as an invalid account, QOS, or GRES. Fix the workload or the Slurm configuration.WaitingForSlurm(Normal): a Slurm limit or partition state holds the job, such as a submit limit or a down partition. The Pods wait until it clears.SlurmRegistrationFailed(Warning): any other error, including an unreachable Slurm.