LeaderWorkerSet

Table of Contents

Overview

Warning

Implicit LeaderWorkerSet gang scheduling is deprecated in slurm-bridge and will be removed in a future release. Admission warns for LWS Pods without spec.schedulingGroup.podGroupName. Enable native LWS gang scheduling with spec.scheduling: {} on the LeaderWorkerSet to migrate.

To reject new legacy LWS Pods and updates that introduce implicit LWS gang scheduling, set admission.featureGates.LegacyLWSGangs: false (default: true). Pods already using implicit LWS gang scheduling can still be updated, with a deprecation warning, so metadata changes and finalizer removal remain possible.

This guide assumes LeaderWorkerSet is installed.

LeaderWorkerSet groups will be co-scheduled so pods of each group will be guaranteed to launch together.

Use spec.groupIdentity: Ordinal (the LWS default) and spec.startupPolicy: LeaderCreated for this integration. LeaderReady is rejected for groups larger than one pod: workers are not created until the leader is ready, but slurm-bridge waits for the full group before scheduling its leader. Each replica group receives its own Slurm allocation, with a default job name of <set-name>-<group-index>. Existing manifests that omit groupIdentity retain ordinal behavior with LWS v1.0.0. The Hash identity mode is not currently supported by slurm-bridge.

The development installer uses LWS v1.0.0. On clusters serving the scheduling.k8s.io/v1beta1 Workload and PodGroup APIs (Kubernetes 1.37), it also enables featureGates.WorkloadAwareScheduling=true and gangSchedulingManagement.schedulerProvider=kubernetes on the LWS controller. Other clusters retain the default LWS configuration.

For native gang scheduling, enable Kubernetes GenericWorkload and the scheduling.k8s.io/v1beta1 API, keep the bridge’s GenericWorkload gate enabled, and configure LWS with the settings above. Set spec.scheduling: {} when creating the LeaderWorkerSet, as in hack/examples/lws/native.yaml. LWS creates one PodGroup per replica and sets each Pod’s spec.schedulingGroup.podGroupName. Each replica receives an independent Slurm allocation named after its PodGroup. Do not set spec.schedulingGroup in the Pod templates when using spec.scheduling.

Existing LeaderWorkerSets cannot add or remove spec.scheduling; recreate the workload to migrate. Workloads without spec.scheduling continue using the deprecated direct scheduling integration, even when the LWS native scheduling gate is enabled. The bridge’s built-in PodGroup integration also supports manually assigned PodGroup references on Pod templates.

When upgrading an existing Helm installation, apply the updated LWS CRDs before upgrading the controller, following the upstream upgrade instructions.

Note

Topology-aware placement is not supported yet, so some features of LeaderWorkerSet may not behave as expected.

See Workloads for shared scheduler and resource configuration.

Scaling

In-place Pod CPU and memory changes are unsupported. Updating the leader or worker resource template triggers the LWS controller’s rollout, replacing groups according to its rollout strategy. Replacement groups request new Slurm allocations; the bridge does not resize the old allocations. This also applies when the old Pods have not started running yet.

Changing spec.leaderWorkerTemplate.size rolls out replacement groups both before and after Slurm allocation. It does not resize an existing queued or running group in place.

Changing spec.replicas adds or removes whole independent groups, each with its own Slurm job. New groups queue independently; removed groups release their allocations during asynchronous cleanup. Retained groups keep their allocations. Removing only some Pods from a shared job retains its full allocation while other members remain active.

The quorum for an implicit LWS gang is the configured group size. If templates put multiple LWS replica groups into one Gang PodGroup, those groups share an allocation and cannot scale independently. Follow the gang scaling limits for that allocation.

Failures and recovery

LWS’s spec.leaderWorkerTemplate.restartPolicy controls group recovery; the default RecreateGroupOnPodRestart recreates the group on Pod recreation or container restart. Replacement Pods need scheduling again. Do not rely on individual Pod replacement to extend or repair an allocated gang; recreate the whole group for a fresh allocation. slurm-bridge supports only RecreateGroupOnPodRestart: Pods from a LeaderWorkerSet with any other restartPolicy, including None, are rejected. When any Pod in a group is lost, slurm-bridge cancels the group’s Slurm job without waiting for the other Pods to stop. LWS then recreates the group, which gets a new Slurm job.

Both spec.podTerminationPolicy values work with slurm-bridge. With the default Sequential, LWS waits for the leader to disappear before deleting its workers. Once the leader stops, it counts as a lost Pod, so the bridge cancels the group’s Slurm job and releases the leader, and LWS then deletes the workers.

With Parallel, LWS starts terminating the leader and workers concurrently. Workloads must tolerate concurrent shutdown. This field is immutable; recreate existing LeaderWorkerSets to change the policy.

See Scaling and failure behavior for shared admission, Slurm update, cleanup, and troubleshooting behavior.