slurm-operator

Download the slurm-operator repository here, start using the operator with a quickstart guide, or read on to learn more.

Slurm and Kubernetes are workload managers originally designed for different kinds of workloads. Kubernetes excels at scheduling workloads that run for an indefinite amount of time, with potentially vague resource requirements, on a single node, with loose policy, but can scale its resource pool infinitely to meet demand; Slurm excels at quickly scheduling workloads that run for a finite amount of time, with well defined resource requirements and topology, on multiple nodes, with strict policy, and a known resource pool.

This project enables the best of both workload managers, unified on Kubernetes.

Why you need slurm-operator and what it can do

slurm-operator allows users to run workloads on Slurm within a Kubernetes cluster, taking advantage of many of the advanced scheduling features of Slurm within a cloud-native environment. This allows for more advanced scheduling than could be achieved within Kubernetes alone, and allows for advanced policies for job priority and allocations, pre-emption, and accounting.

It contains a Kubernetes operator to deploy and manage certain components of Slurm clusters. This repository implements custom-controllers and custom resource definitions (CRDs) designed for managing the lifecycle of Slurm clusters.

Image

For additional architectural notes, see the architecture docs.

Features

management of Slurm clusters on Kubernetes. The operator takes into consideration the running workload among Slurm nodes as it needs to scale-in, upgrade, or otherwise handle node failures. Slurm nodes are marked as drain before their eventual termination pending scale-in or upgrade.

Slurm

Slurm is a full featured HPC workload manager. To highlight a few features:

  • Accounting: collect accounting information for every job and job step executed.
  • Partitions: job queues with sets of resources and constraints (e.g. job size limit, job time limit, users permitted).
  • Reservations: reserve resources for jobs being executed by select users and/or select accounts.
  • Job Dependencies: defer the start of jobs until the specified dependencies have been satisfied.
  • Job Containers: jobs which run an unprivileged OCI container bundle.
  • Priority: assigns priorities to jobs upon submission and on an ongoing basis (e.g. as they age).
  • Preemption: stop one or more low-priority jobs to let a high-priority job run.
  • QoS: sets of policies affecting scheduling priority, preemption, and resource limits.
  • Fairshare: distribute resources equitably among users and accounts based on historical usage.
  • Node Health Check: periodically check node health via script.

Quickstart

See the quickstart guide for instructions on installing slurm-operator.

Supported Versions

  • Kubernetes Version >= v1.29
  • Slurm Version >= 24.11

Current Limitations

  • cgroup v2 is the only supported cgroup implementation for Slinky

Upgrades

In 0.X.0 releases, breaking changes may be introduced into newer CRDs. To upgrade between these versions, uninstall all Slinky charts and delete Slinky CRDs, then install the new release like normal.

helm --namespace=slurm uninstall slurm
helm --namespace=slinky uninstall slurm-operator
kubectl delete clusters.slinky.slurm.net
kubectl delete nodesets.slinky.slurm.net

License

Copyright (C) SchedMD LLC.

Licensed under the Apache License, Version 2.0 you may not use project except in compliance with the license.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

0.1.x

0.2.x

0.3.x