Development

Local Kind cluster

Create a development cluster and deploy the operator, CRDs, and a Slurm cluster:

make kind-start

Use make deploy for a one-time rebuild or make debug for Delve-enabled images. Delete the cluster with:

make kind-stop

The generated helm/*/values-dev.yaml files are sparse, untracked overrides. Add only values needed for development. Skaffold resets each release to the current chart defaults before applying these overrides. If an older file is a full copy of values.yaml, replace it with {} so it does not pin defaults from an older checkout.

Scale testing with KWOK

KWOK runs pods on fake nodes without starting containers, so a laptop can hold thousands of NodeSet pods. Create a cluster with the operator and KWOK, then add 100 fake nodes:

./hack/kind.sh --crds --operator --kwok
./hack/kwok-nodes.sh

Each fake node fits 110 pods. For more nodes, run ./hack/kwok-nodes.sh COUNT with the total you need.

Apply the test NodeSet, then scale it:

kubectl apply -f hack/resources/kwok-nodeset.yaml
kubectl scale nodeset/slurm-kwok --replicas=1000
kubectl get nodeset/slurm-kwok --watch

The fixture has no RestApi, so the operator skips its Slurm calls and only the Kubernetes side is exercised. KWOK’s fast stages mark those pods Ready without starting slurmd.

To exercise intermittent Kubernetes and slurmrestd errors, install the restapi mock after the KWOK fixture. It is a Go slurmrestd the operator dials on port 6820. Node state is kept in the slurm-client fake client and filled from the live slurmd pods. Requests can fail or be delayed by a fixed latency with random jitter. Counts and average latency are served at GET /restapi/stats on that port. The same pod serves a Kubernetes admission webhook that can deny slurmd pod creates.

./hack/kwok-restapi.sh --slurm=0 --kube=0
./hack/kwok-restapi.sh --slurm=0.05 --kube=0.05 --delay=20ms --jitter=5ms
kubectl port-forward svc/slurm-kwok-restapi 6820:6820
curl -s localhost:6820/restapi/stats

Rates are fractions from 0 to 1. --slurm fails HTTP calls to the mock. --kube denies pod creates. --delay is the fixed request latency. --jitter randomly deviates from that delay in either direction, clamped at zero. Re-run the script to change them. Scale the NodeSet back to 0 between runs.

To remove the fake nodes without deleting the cluster, run:

kubectl delete nodes --selector=app.kubernetes.io/managed-by=slurm-operator-kwok

Remote cluster

Update an existing Slinky installation

Install a compatible released Slinky stack first. The workstation running the deployment needs Make, Skaffold, Helm, Docker, Kubernetes API access, and push access to a registry that the cluster can pull from.

Select the Kubernetes context and registry, then deploy from the checkout:

export SKAFFOLD_KUBE_CONTEXT=my-remote-cluster
export SKAFFOLD_DEFAULT_REPO=ghcr.io/my-user
make deploy

Skaffold builds and pushes the operator images, then Helm upgrades the CRDs and operator using the current chart defaults and values-dev.yaml overrides. Existing release values are not retained, so add required cluster-specific values to values-dev.yaml before deploying. It does not upgrade the Slurm chart. To test changes to that chart, upgrade it explicitly with the environment’s values:

helm upgrade slurm ./helm/slurm \
  --kube-context my-remote-cluster \
  --namespace slurm \
  --reset-values \
  --values ./helm/slurm/values-dev.yaml \
  --wait

Bootstrap the current Kubernetes context

To install cert-manager, the CRDs, the locally built operator, and Slurm onto an existing Kubernetes cluster, select its context and provide a registry that the cluster can pull from:

kubectl config use-context my-remote-cluster
./hack/kind.sh \
  --existing-cluster \
  --registry=ghcr.io/my-user \
  --core

--registry sets SKAFFOLD_DEFAULT_REPO; setting that environment variable directly is equivalent. This workflow requires Make, Skaffold, Helm, kubectl, Go, yq, Docker or Podman, and push access to the registry.

To install only the external prerequisite, cert-manager, run:

make prereqs

The prerequisites-only command requires Helm and kubectl, but does not require Skaffold or a local container build toolchain.