Controllers

Table of Contents

Overview

The scheduler process runs two controllers:

  • Node Controller - Registers and taints managed Kubernetes nodes.

  • Registration Controller - Creates Slurm allocations, publishes Pod registration metadata, and cleans up completed workloads.

Both are enabled by default.

When upgrading from separate deployments, move controllers resource, placement, and registrationWorkers overrides to scheduler. The Helm pre-upgrade hook stops the old components; if upgradeHook.enabled=false, stop them manually before upgrading.

The chart always enables leader election. Standalone binaries use --leader-elect; KubeSchedulerConfiguration.leaderElection.leaderElect and the old scheduler.leaderElect Helm value are ignored. The lease uses leaderElection.resourceNamespace from the scheduler configuration (default: kube-system). Override it with --leader-election-namespace. Helm uses the release namespace.

Node Controller

The node controller taints managed nodes and synchronizes their scheduling state with Slurm.

A managed node is a Kubernetes node with a corresponding Slurm node. This includes both external and hybrid nodes; only hybrid nodes require a colocated slurmd.

Node Registration and Partition Assignment

The node controller supports dynamic node registration in Slurm through Kubernetes node labels:

  • Node Registration Label (scheduler.slinky.slurm.net/external-node): When this label is present on a Kubernetes node, the node controller will automatically register the node in Slurm with the correct CPU and memory resources. When the label is removed, the node will be removed from Slurm.

  • Node Partitions Annotation (scheduler.slinky.slurm.net/external-node-partitions): Specifies the Slurm partition(s) the node should be assigned to. The value is a comma-separated list of partition names. When this annotation is set, the node controller will:

    1. Validate that all specified partitions exist in Slurm

    2. Register the node with node features matching the partition names

    3. Allow Slurm’s NodeSet configuration in slurm.conf to assign the node to those partitions

    Example:

    apiVersion: v1
    kind: Node
    metadata:
      name: worker-1
      labels:
        scheduler.slinky.slurm.net/external-node: "true"
      annotations:
        scheduler.slinky.slurm.net/external-node-partitions: "slurm-bridge,gpu"
    

    This will register the node with features slurm-bridge,gpu, allowing it to be assigned to those partitions in slurm.conf using NodeSet features.

    For external nodes, the bridge owns the Slurm node’s features: they are the partitions above, plus the features in the feature.slinky.slurm.net/spec annotation, plus the bridge’s own compatibility feature. A feature removed from either annotation is removed from the Slurm node, and features set directly in Slurm are replaced. Hybrid nodes (those registered by their own slurmd) keep the features and topology they registered with.

    Note: The controller will error if any specified partition does not exist in Slurm. Partitions must be pre-configured in Slurm’s configuration.

sequenceDiagram autonumber participant KAPI as Kubernetes API participant SWC as Node Controller participant SAPI as Slurm REST API loop Reconcile Loop KAPI-->>SWC: Watch Kubernetes Nodes alt Node is managed SWC->>KAPI: Taint Node KAPI-->>SWC: Return Status else SWC->>KAPI: Untaint Node KAPI-->>SWC: Return Status end %% alt Node is managed alt Node is unschedulable SWC->>SAPI: Drain Node SAPI-->>SWC: Return Status else SWC->>SAPI: Undrain Node SAPI-->>SWC: Return Status end %% alt Node is unschedulable end %% loop Reconcile Loop

Registration Controller

The registration controller creates held Slurm allocations and removes Pod registration gates once allocation metadata is published. It also repairs metadata and cleans up completed allocations.

Cleanup retains recorded, published, or bound members. Before cancellation it confirms membership through the Kubernetes API and ownership through Slurm. A deleting, bound Pod retains its allocation until it becomes terminal. A Pod on a node with no working kubelet never does; once its containers have stopped, taint the node with node.kubernetes.io/out-of-service=nodeshutdown:NoExecute. Pod garbage collection then marks the Pod Failed and cleanup releases its job. Running heterogeneous jobs can release individual nonzero components. Component zero and pending heterogeneous jobs are retained while any member needs the allocation.

Bound Pods whose jobs stop running are deleted after a live Slurm check. Claims and scheduler finalizers are removed only after Slurm reports the allocation finished or absent; a cancellation acknowledgement alone does not release them.

Configure concurrency with scheduler.registrationWorkers in Helm or --registration-workers on the combined scheduler binary. The default is 10 workers. The Node controller also defaults to 10 workers, configurable with --node-workers. scheduler.podWorkers and --pod-workers remain accepted for compatibility but have no effect.

Submission Failures

When Slurm refuses a submission, the Pods stay behind their registration gate. The controller records an Event with Slurm’s message on the Pod or its group object, such as a PodGroup, and retries later:

  • SlurmRegistrationRejected (Warning): Slurm refused the request itself, such as an invalid account, QOS, or GRES. Fix the workload or the Slurm configuration.

  • WaitingForSlurm (Normal): a Slurm limit or partition state holds the job, such as a submit limit or a down partition. The Pods wait until it clears.

  • SlurmRegistrationFailed (Warning): any other error, including an unreachable Slurm.