Developing Slurm Bridge
Local Kind cluster
The workstation needs Make, Docker, Kind, Skaffold, Helm, kubectl, and Go. Create a development cluster and deploy the complete bridge stack:
make kind-start
Kind 0.33.0 is required. The script selects a digest-pinned Kubernetes 1.36 node image and the matching Kind configuration, independently of Kind’s default image. Inspect the selected image without creating a cluster:
make kind-image
Select Kubernetes 1.35 with its matching configuration:
KUBERNETES_VERSION=v1.35 ./hack/kind.sh --core slurm-bridge-1-35
KUBERNETES_VERSION accepts a minor selector such as v1.36 or a patch
selector such as v1.36.0; both select the pinned image for that minor. Use
KIND_NODE_IMAGE to select a specific image and KIND_CONFIG (or --config)
to override the configuration. Other Kubernetes minors require both overrides.
Image selection only affects newly created clusters; use a new cluster name or
--recreate to replace an existing cluster (which deletes its workloads).
Use make deploy for a one-time rebuild. Delete the cluster with:
make kind-stop
The generated helm/slurm-bridge/values-dev.yaml file is a sparse, untracked
override. Add only values needed for development. Skaffold resets the release to
the current chart defaults before applying these overrides. If an older file is
a full copy of values.yaml, replace it with {} so it does not pin defaults
from an older checkout.
End-to-end tests
The end-to-end suite can exercise either supported Slurm node mode. External mode is the default:
make kind-start test-e2e
Use a separate cluster for hybrid mode because an installed cluster cannot switch node modes in place:
KIND_CLUSTER_NAME=slurm-bridge-hybrid \
SLURM_NODE_MODE=hybrid \
make kind-start test-e2e
SLURM_NODE_MODE is also passed into the test process. The readiness feature
uses it to verify that the cluster actually contains external nodes or
DaemonSet-mode hybrid slurmd pods, as requested. Hybrid runs also include a
native sbatch feature labeled slurm-node-mode=hybrid, which verifies that a
job submitted directly to Slurm completes on one of those hybrid workers.
After the serial readiness check, independent workload features run in parallel. Slurm may queue jobs when the suite temporarily asks for more nodes than are available, so each feature allows up to ten minutes for an allocation.
Use E2E_RUN with a Go test regular expression to select one feature. Set
E2E_CLEANUP=false to leave the successful workload in place for debugging:
MOCK_NVML=true \
E2E_RUN='TestScheduling/Non-exclusive_DRA_resources_allocated_to_container$' \
E2E_CLEANUP=false \
make kind-start test-e2e
The cluster is never removed by test-e2e; make kind-stop remains explicit.
Cancellation features still remove their workload because deletion is the
behavior they validate.
Release 1.2 covers Jobs, JobSets, native and scheduler-plugins PodGroups, LeaderWorkerSets, admission routing, Slurm annotations, cancellation, and the CPU and GPU DRA drivers. DRANET requires configurable device profiles and is not part of this release’s suite. Hybrid fixtures use DeviceClass names as Slurm GRES types, as required by the release 1.2 allocator.
Remote cluster
Install a compatible released Slinky stack first. The workstation running the deployment needs Kubernetes API access, push access to a registry, and a registry namespace that the cluster can pull from.
Select the Kubernetes context and registry, then deploy the local checkout:
export SKAFFOLD_KUBE_CONTEXT=my-remote-cluster
export SKAFFOLD_DEFAULT_REPO=ghcr.io/my-user
make deploy
Skaffold builds and pushes the bridge images, then Helm upgrades Slurm Bridge
using the current chart defaults and values-dev.yaml overrides. Existing
release values are not retained, so add any required cluster-specific values to
values-dev.yaml before deploying.
Specialized fixtures
Install all optional Kind development fixtures:
./hack/kind.sh --extras slurm-bridge-dev
This installs the CPU, example GPU, and NVIDIA GPU DRA drivers. Each driver can
still be installed individually with --dra-driver-cpu, --dra-example-driver,
or --dra-driver-nvidia-gpu.
NVIDIA GPU DRA normally requires GPU-equipped workers with the NVIDIA driver
installed on the host. For local testing without GPUs, set MOCK_NVML=true to
install
nvml-mock
on the managed Kind workers before the NVIDIA DRA driver:
MOCK_NVML=true make kind-start
Examples remain individually selectable:
kubectl apply -f hack/examples/job/single.yaml
kubectl apply -f hack/examples/dra/gpu-example/job.yaml
Demo
Create the core stack, install all optional fixtures, and run a curated set of finite example workloads:
make demo-start
Watch the workloads and their corresponding Slurm jobs until interrupted with
Ctrl+C:
./hack/watch.sh --demo
For an annotated view, or an annotated split-screen view with tmux, run:
./hack/watch.sh --explain
./hack/watch.sh --explain --tmux
The watchers require watch; the split-screen mode also requires tmux.
Stopping a watcher does not remove the demo workloads. Remove them with:
make demo-stop
The cluster remains available for development. Delete it with make kind-stop.
Optional system diagnostics remain a direct command:
sudo ./hack/sysctl.sh