Quickstart

Overview

This quickstart guide will help you get the slurm-bridge running and configured with your existing Slurm cluster.

Slurm Configuration

There are a set of assumptions that the slurm-bridge-scheduler must make. The Slurm admin must satisfy those assumption in their Slurm cluster.

There exists a set of hosts that have colocated kubelet and slurmd (installed on the same host, typically running as a systemd service).
Slurm is configured with a partition with that only contains host with colocated kubelet and slurmd.
- The partition name must match the one configured in values.yaml used to deploy the slurm-bridge helm chart (default: “slurm-bridge”).
- Example slurm.conf snippet.
```
# slurm.conf
...
NodeSet=kubernetes Feature=kubernetes
PartitionName=slurm-bridge Nodes=kubernetes
```
In the event that the colocated node’s Slurm NodeName does not match the Kubernetes Node name, you should patch the Kubernetes node with a label to allow slurm-bridge to map the colocated Kubernetes and Slurm node.
```
kubectl patch node $KUBERNETES_NODENAME -p "{\"metadata\":{\"labels\":{\"slinky.slurm.net/slurm-nodename\":\"$SLURM_NODENAME\"}}}"
```

Slurm has Multi-Category Security enabled for labels.

Example slurm.conf snippet.

# slurm.conf
...
MCSPlugin=mcs/label
MCSParameters=ondemand,ondemandselect

Install `slurm-bridge`

Pre-Requisites

Install the pre-requisite helm charts.

helm repo update
helm install cert-manager jetstack/cert-manager \
	--namespace cert-manager --create-namespace --set crds.enabled=true

Slurm Bridge

Download values and install the slurm-bridge from OCI package:

curl -L https://raw.githubusercontent.com/SlinkyProject/slurm-bridge/refs/tags/v0.2.0/helm/slurm-bridge/values.yaml \
  -o values-bridge.yaml
helm install slurm-bridge oci://ghcr.io/slinkyproject/charts/slurm-bridge \
  --values=values-bridge.yaml --version=0.3.0 --namespace=slinky --create-namespace

You can check if your cluster deployed successfully with:

kubectl --namespace=slinky get pods

Your output should be similar to:

NAME                                        READY   STATUS    RESTARTS      AGE
slurm-bridge-admission-85f89cf884-8c9jt     1/1     Running   0             1m0s
slurm-bridge-controllers-757f64b875-bsfnf   1/1     Running   0             1m0s
slurm-bridge-scheduler-5484467f55-wtspk     1/1     Running   0             1m0s

Scheduling Workload

Generally speaking, slurm-bridge translates one or more pods into a representative Slurm workload, where Slurm does the underlying scheduling. Certain optimizations can be made, depending on which resource is being translated.

slurm-bridge has specific scheduling support for JobSet and PodGroup resources and their pods. If your workload requires or benefits from co-scheduled pod launch (e.g. MPI, multi-node), consider representing your workload as a JobSet or PodGroup.

Admission Controller

slurm-bridge will only schedule pods who requests slurm-bridge as its scheduler. The slurm-bridge admission controller can be configured to automatically make slurm-bridge the scheduler for all pods created in the configured namespaces.

Alternatively, a pod can specify Pod.Spec.schedulerName=slurm-bridge-scheduler from any namespace.

Annotations

Users can better inform or influence slurm-bridge how to represent their Kubernetes workload within Slurm by adding annotations on the parent Object.

Example “pause” bare pod to illustrate annotations:

apiVersion: v1
kind: Pod
metadata:
  name: pause
  # slurm-bridge annotations on parent object
  annotations:
    slinky.slurm.net/timelimit: "5"
    slinky.slurm.net/account: foo
spec:
  schedulerName: slurm-bridge-scheduler
  containers:
    - name: pause
      image: registry.k8s.io/pause:3.6
      resources:
        limits:
          cpu: "1"
          memory: 100Mi

Example “pause” deployment to illustrate annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pause
  # slurm-bridge annotations on parent object
  annotations:
    slinky.slurm.net/timelimit: "5"
    slinky.slurm.net/account: foo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: pause
  template:
    metadata:
      labels:
        app: pause
    spec:
      schedulerName: slurm-bridge-scheduler
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.6
          resources:
            limits:
              cpu: "1"
              memory: 100Mi

JobSets

This section assumes JobSets is installed.

JobSet pods will be coscheduled and launched together. The JobSet controller is responsible for managing the JobSet status and other Pod interactions once marked as completed.

PodGroups

This section assumes PodGroups CRD is installed and the out-of-tree kube-scheduler is installed and configured as a (non-primary) scheduler.

Pods contained within a PodGroup will be co-scheduled and launched together. The PodGroup controller is responsible for managing the PodGroup status and other Pod interactions once marked as completed.