Developing Slurm Bridge
Local Kind cluster
The workstation needs Make, Docker, Kind, Skaffold, Helm, kubectl, and Go. Create a development cluster and deploy the complete bridge stack:
make kind-start
Use make deploy for a one-time rebuild or make debug for Delve-enabled
images. Delete the cluster with:
make kind-stop
The generated helm/slurm-bridge/values-dev.yaml file is a sparse, untracked
override. Add only values needed for development. Skaffold resets the release to
the current chart defaults before applying these overrides. If an older file is
a full copy of values.yaml, replace it with {} so it does not pin defaults
from an older checkout.
End-to-end tests
The end-to-end suite can exercise either supported Slurm node mode. External mode is the default:
make kind-start test-e2e
Use a separate cluster for hybrid mode because an installed cluster cannot switch node modes in place:
KIND_CLUSTER_NAME=slurm-bridge-hybrid \
SLURM_NODE_MODE=hybrid \
make kind-start test-e2e
Select another supported Kubernetes version with KUBERNETES_VERSION:
KIND_CLUSTER_NAME=slurm-bridge-1-36 \
KUBERNETES_VERSION=v1.36 \
make kind-start test-e2e
The suite only talks to the kubeconfig context named by E2E_KUBE_CONTEXT,
which defaults to kind-$KIND_CLUSTER_NAME. It never falls back to the current
context. Set it explicitly to test a cluster that was not created by
kind-start.
SLURM_NODE_MODE is also passed into the test process. The readiness feature
uses it to verify that the cluster actually contains external nodes or
DaemonSet-mode hybrid slurmd pods, as requested. Hybrid runs also include a
native sbatch feature labeled slurm-node-mode=hybrid, which verifies that a
job submitted directly to Slurm completes on one of those hybrid workers.
After the serial readiness check, independent workload features run in parallel. Slurm may queue jobs when the suite temporarily asks for more nodes than are available, so each feature allows up to ten minutes for an allocation.
CPU DRA installation and tests require Kubernetes 1.36 or newer. On 1.35, both node modes still test native CPU scheduling and GPU/NIC DRA.
Run the focused CPU/GPU allocation, exclusive-request rejection, and CPU reuse checks on an existing development cluster with:
E2E_RUN=TestSocketCPUAllocation make test-e2e
These checks compare Slurm’s CPU bitmap with the driver’s allocation and the container’s effective CPU set. They do not cover driver restarts or host reboots.
Use E2E_RUN with a Go test regular expression to select one feature. Set
E2E_CLEANUP=false to leave the successful workload in place for debugging:
MOCK_NVML=true \
E2E_RUN='TestScheduling/Non-exclusive_DRA_resources_allocated_to_container$' \
E2E_CLEANUP=false \
make kind-start test-e2e
The cluster is never removed by test-e2e; make kind-stop remains explicit.
Cancellation features still remove their workload because deletion is the
behavior they validate.
Ray-generated PodGroups
The Ray scheduling feature requires a running KubeRay operator with
KubernetesWAS=true, the ray.io/v1 RayJob and RayCluster CRDs, and a native
Workload/PodGroup API (Kubernetes 1.36 or newer). Install the optional operator
fixture with ./hack/kind.sh --extras slurm-bridge-dev, or install only
KubeRay:
./hack/kind.sh --kuberay slurm-bridge-dev
Kubernetes 1.37 builds the unmodified upstream revision pinned in
hack/kuberay/build.sh, loads it into Kind, and
installs its matching chart with native v1alpha3 scheduling enabled. The
released KubeRay 1.7.1 provider only supports v1alpha2, so older clusters use
that release with native scheduling enabled when v1alpha2 is served. On
Kubernetes 1.35 the operator is installed without native scheduling, and the Ray
feature skips. For Kubernetes 1.37 with --existing-cluster, set
KUBERAY_IMAGE to a tagged build of the pinned revision available to the
cluster. Use only native WAS, without an external batchScheduler integration.
With KubeRay and Slurm Bridge deployed in the selected cluster, run:
E2E_KUBE_CONTEXT=kind-slurm-bridge-dev \
E2E_LABELS=suite=scheduling \
E2E_RUN='^TestScheduling$/^RayJob_generated_native_PodGroup_scheduling$' \
make test-e2e
The feature creates an HTTP-mode RayJob with one head and one worker. KubeRay
must create its RayCluster and native PodGroup; the test does not create either
on its behalf. It selects the head and worker using a unique label and checks
that they run on two bridge nodes sharing one Slurm allocation, reference the
same native PodGroup, and have a scheduled PodGroup. Deleting the RayJob must
release its allocation. HTTP submission avoids requiring an additional node for
a separate submitter Job. The Ray images are rayproject/ray:2.52.0, with 3 GiB
for the head and 1 GiB for the worker. This feature runs in the scheduling suite
and demonstrates basic scheduling, without testing Ray autoscaling, recovery, or
job execution results.
Remote cluster
Install a compatible released Slinky stack first. The workstation running the deployment needs Kubernetes API access, push access to a registry, and a registry namespace that the cluster can pull from.
Select the Kubernetes context and registry, then deploy the local checkout:
export SKAFFOLD_KUBE_CONTEXT=my-remote-cluster
export SKAFFOLD_DEFAULT_REPO=ghcr.io/my-user
make deploy
Skaffold builds and pushes the bridge images, then Helm upgrades Slurm Bridge
using the current chart defaults and values-dev.yaml overrides. Existing
release values are not retained, so add any required cluster-specific values to
values-dev.yaml before deploying.
Specialized fixtures
Install all optional DRA and KubeRay Kind development fixtures:
./hack/kind.sh --extras slurm-bridge-dev
This installs the CPU (Kubernetes 1.36+), example GPU, NVIDIA GPU, Google TPU,
and DRANET DRA drivers, plus the KubeRay operator described above. Each driver
can still be installed individually with --dra-driver-cpu,
--dra-example-driver, --dra-driver-nvidia-gpu, --dra-driver-google-tpu, or
--dranet. The --dranet fixture creates a dranet0 dummy interface on each
managed Kind worker before installing the driver.
The CPU fixture builds the unmodified upstream revision pinned in
hack/dra-driver-cpu/build.sh, loads it into
Kind, and installs its matching chart with
values.yaml. For --existing-cluster,
set DRA_CPU_IMAGE to a tagged build of that revision that every node can pull.
Hosts without usable NUMA sysfs need the driver’s sysfsOverlay test fixture,
which MOCK_NVML=true applies; synthetic topology does not validate physical
socket or SMT discovery.
The --dra-example-driver, --dranet, and --dra-driver-google-tpu flags also
configure Slurm Bridge with the
e2e device profiles. This explicitly enables
the example GPU, dummy network interface, and TPU profiles alongside the
built-in profiles. The example GPU profile is disabled by default.
NVIDIA GPU DRA normally requires GPU-equipped workers with the NVIDIA driver
installed on the host. For local testing without GPUs, set MOCK_NVML=true to
install
nvml-mock
on the managed Kind workers before the NVIDIA DRA driver:
MOCK_NVML=true make kind-start
MOCK_NVML=true also mocks two NUMA domains sharing one physical socket for
NUMA alignment tests: GPUs 0-3 and CPUs 0-3 on
NUMA node 0, and GPUs 4-7, CPUs 4-7, and dranet0 on NUMA node 1. The CPU
topology is mocked only for external Slurm nodes, which need at least eight
CPUs, so the NUMA alignment test skips hybrid mode. The test verifies that the
inventory represents sub-NUMA clustering, Slurm registers two sockets with four
cores each, and GPU/NIC socket affinity is retained before checking the job’s
allocation and effective container CPU set.
Run this regression on the mocked Kubernetes 1.37+ cluster with:
MOCK_NVML=true E2E_RUN=TestNUMAAlignment make test-e2e
Examples remain individually selectable:
kubectl apply -f hack/examples/job/single.yaml
kubectl apply -f hack/examples/dra/dranet/job.yaml
kubectl apply -f hack/examples/dra/gpu-example/job.yaml
kubectl apply -f hack/examples/dra/nvidia/job.yaml
Demo
Create the core stack, install all optional fixtures, and run a curated set of finite example workloads:
make demo-start
Watch the workloads and their corresponding Slurm jobs until interrupted with
Ctrl+C:
./hack/watch.sh --demo
For an annotated view, or an annotated split-screen view with tmux, run:
./hack/watch.sh --explain
./hack/watch.sh --explain --tmux
The watchers require watch; the split-screen mode also requires tmux.
Stopping a watcher does not remove the demo workloads. Remove them with:
make demo-stop
The cluster remains available for development. Delete it with make kind-stop.
For the fake TPU fixture and allocation/reuse test, see TPU DRA.
Optional system diagnostics remain a direct command:
sudo ./hack/sysctl.sh