Development
Local Kind cluster
Create a development cluster and deploy the operator, CRDs, and a Slurm cluster:
make kind-start
Use make deploy for a one-time rebuild or make debug for Delve-enabled
images. Delete the cluster with:
make kind-stop
The generated helm/*/values-dev.yaml files are sparse, untracked overrides.
Add only values needed for development. Skaffold resets each release to the
current chart defaults before applying these overrides. If an older file is a
full copy of values.yaml, replace it with {} so it does not pin defaults
from an older checkout.
Scale testing with KWOK
KWOK runs pods on fake nodes without starting containers, so a laptop can hold thousands of NodeSet pods. Create a cluster with the operator and KWOK, then add 100 fake nodes:
./hack/kind.sh --crds --operator --kwok
./hack/kwok-nodes.sh
Each fake node fits 110 pods. For more nodes, run ./hack/kwok-nodes.sh COUNT
with the total you need.
Apply the test NodeSet, then scale it:
kubectl apply -f hack/resources/kwok-nodeset.yaml
kubectl scale nodeset/slurm-kwok --replicas=1000
kubectl get nodeset/slurm-kwok --watch
The fixture has no RestApi, so the operator skips its Slurm calls and only the Kubernetes side is exercised. KWOK’s fast stages mark those pods Ready without starting slurmd.
To exercise intermittent Kubernetes and slurmrestd errors, install the restapi
mock after the KWOK fixture. It is a Go slurmrestd the operator dials on port
6820. Node state is kept in the slurm-client fake client and filled from the
live slurmd pods. Requests can fail or be delayed by a fixed latency with random
jitter. Counts and average latency are served at GET /restapi/stats on that
port. The same pod serves a Kubernetes admission webhook that can deny slurmd
pod creates.
./hack/kwok-restapi.sh --slurm=0 --kube=0
./hack/kwok-restapi.sh --slurm=0.05 --kube=0.05 --delay=20ms --jitter=5ms
kubectl port-forward svc/slurm-kwok-restapi 6820:6820
curl -s localhost:6820/restapi/stats
Rates are fractions from 0 to 1. --slurm fails HTTP calls to the mock.
--kube denies pod creates. --delay is the fixed request latency. --jitter
randomly deviates from that delay in either direction, clamped at zero. Re-run
the script to change them. Scale the NodeSet back to 0 between runs.
To remove the fake nodes without deleting the cluster, run:
kubectl delete nodes --selector=app.kubernetes.io/managed-by=slurm-operator-kwok
Remote cluster
Update an existing Slinky installation
Install a compatible released Slinky stack first. The workstation running the deployment needs Make, Skaffold, Helm, Docker, Kubernetes API access, and push access to a registry that the cluster can pull from.
Select the Kubernetes context and registry, then deploy from the checkout:
export SKAFFOLD_KUBE_CONTEXT=my-remote-cluster
export SKAFFOLD_DEFAULT_REPO=ghcr.io/my-user
make deploy
Skaffold builds and pushes the operator images, then Helm upgrades the CRDs and
operator using the current chart defaults and values-dev.yaml overrides.
Existing release values are not retained, so add required cluster-specific
values to values-dev.yaml before deploying. It does not upgrade the Slurm
chart. To test changes to that chart, upgrade it explicitly with the
environment’s values:
helm upgrade slurm ./helm/slurm \
--kube-context my-remote-cluster \
--namespace slurm \
--reset-values \
--values ./helm/slurm/values-dev.yaml \
--wait
Bootstrap the current Kubernetes context
To install cert-manager, the CRDs, the locally built operator, and Slurm onto an existing Kubernetes cluster, select its context and provide a registry that the cluster can pull from:
kubectl config use-context my-remote-cluster
./hack/kind.sh \
--existing-cluster \
--registry=ghcr.io/my-user \
--core
--registry sets SKAFFOLD_DEFAULT_REPO; setting that environment variable
directly is equivalent. This workflow requires Make, Skaffold, Helm, kubectl,
Go, yq, Docker or Podman, and push access to the registry.
To install only the external prerequisite, cert-manager, run:
make prereqs
The prerequisites-only command requires Helm and kubectl, but does not require Skaffold or a local container build toolchain.