GKE with Cloud TPUs and Slurm-operator on JAX
This tutorial guides you through setting up a Slurm cluster on Google Kubernetes Engine (GKE) with Cloud TPU (v5e) support using Slurm-operator. You will deploy the operator, configure TPU GRES support with necessary GKE workarounds, and run a JAX model training test job.
Table of Contents
1. Plan & Create GKE Infrastructure
To get started, you need a GKE cluster (Standard) and a TPU v5e node pool. In this tutorial, we will target a single-host 8-chip TPU v5e node pool.
Set up your environment variables and create the cluster:
PROJECT="<YOUR_GCP_PROJECT_ID>"
CLUSTER="tpu-slurm-cluster"
ZONE="us-central1-a"
VERSION="1.35"
# Create the base GKE Standard cluster
gcloud container clusters create "${CLUSTER}" \
--cluster-version="${VERSION}" \
--zone="${ZONE}" \
--project="${PROJECT}" \
--num-nodes="3"
# Create the TPU v5e Spot node pool (8-chip ct5lp-hightpu-8t)
gcloud container node-pools create tpu-v5-single-host-spot \
--cluster="${CLUSTER}" \
--zone="${ZONE}" \
--project="${PROJECT}" \
--machine-type="ct5lp-hightpu-8t" \
--spot \
--num-nodes="1" \
--node-locations="${ZONE}"
2. Install Operator Prerequisites
The default slurm-operator chart configuration uses cert-manager to manage
admission webhook certificates. The chart can instead generate a self-signed
certificate when installed with certManager.enabled=false.
2.1. Install Cert-Manager
Use a cert-manager release that supports the cluster’s Kubernetes version; see the supported releases.
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--namespace cert-manager --create-namespace \
--set crds.enabled=true
3. Deploy Slurm-operator
We will use the latest stable release of Slurm-operator. A list of releases for Slurm-operator can be found here.
Install the operator with CRDs enabled:
helm install slurm-operator oci://ghcr.io/slinkyproject/charts/slurm-operator \ --set crds.enabled=true \ -n slinky \ --create-namespace
Verify both operator pods are running successfully:
kubectl get pods -n slinky
4. Configure TPU Support in Slurm
We must prepare a custom tpu-values.yaml for the Slurm cluster deployment.
Because GKE and Slurm-operator have specific requirements, we must apply three
critical workarounds:
GKE Warden Topology Alignment: GKE Warden validating webhook requires Pods requesting TPUs to have exact topology selectors matching the node pool. We specify
tpu-v5-lite-podsliceand2x4(8 chips).Manual host
/devmount: GKE TPU v5e uses thevfio-pcidriver (devices exposed via/dev/vfio/*). Since the GKE webhook might skip automatic device mounting for custom slurmd pods, we manually mount/devfrom the host.Count-Based GRES: Because traditional
/dev/accel*files do not exist on VFIO-based TPU v5e nodes, listing file paths ingres.confwill causeslurmdstartup crashes. We configure Slurm GRES by Count only (Name=tpu Count=8), which is safe since device constraints are not enforced by cgroups by default.
Save the following as tpu-values.yaml in your workspace root:
# tpu-values.yaml
clusterName: slurm-tpu-cluster
controller:
# Add TPU to general GRES types
extraConf: |
GresTypes=gpu,tpu
configFiles:
# Use Count-based GRES (no File paths) to avoid slurmd stat crashes.
gres.conf: |
Name=tpu Count=8
nodesets:
tpu:
enabled: true
scalingMode: StatefulSet
replicas: 1 # Deploy 1 worker pod on our single 8-chip node
slurmd:
resources:
limits:
google.com/tpu: 8 # Request all 8 chips on the node
requests:
google.com/tpu: 8
# Mount host /dev to access /dev/vfio/* TPU devices
volumeMounts:
- name: host-dev
mountPath: /dev
extraConfMap:
Gres: ["tpu:8"]
podSpec:
# GKE Warden Topology Requirements
nodeSelector:
cloud.google.com/gke-tpu-accelerator: "tpu-v5-lite-podslice"
cloud.google.com/gke-tpu-topology: "2x4"
tolerations:
- key: "google.com/tpu"
operator: "Exists"
effect: "NoSchedule"
volumes:
- name: host-dev
hostPath:
path: /dev
type: Directory
partitions:
tpu:
enabled: true
nodesets:
- tpu
configMap:
Default: "YES"
MaxTime: "UNLIMITED"
State: "UP"
5. Deploy the Slurm Cluster
Now, deploy the Slurm cluster using Helm:
helm install slurm-tpu oci://ghcr.io/slinkyproject/charts/slurm -f tpu-values.yaml -n slurm --create-namespace
Verify GRES Registration
Once all pods in the slurm namespace are Running, check if Slurmctld has
successfully registered the 8 TPUs:
kubectl exec -n slurm slurm-tpu-controller-0 -c slurmctld -- sinfo -o "%n %G"
Expected output:
HOSTNAMES GRES
tpu-0 tpu:8
The TPU node tpu-0 successfully registers tpu:8!
6. Running a JAX TPU Workload
Note
For a proof-of-concept, this tutorial installs Python system packages and JAX
dynamically at runtime inside the job script. For a production environment, it
is highly recommended to bake JAX and its dependencies directly into the
slurmd container image at build-time to reduce startup latency, improve
reliability, and ensure network isolation. This can be done similarly to the
Nvidia/PyTorch tutorial by extending Slinky’s base
Dockerfile.
Since the default Slinky slurmd image is a minimal Ubuntu image without Python
packages, we will write a submission script that installs python3-pip and
python3-venv inside the container at runtime. In this flow, sbatch is
invoked as root in the worker container, so the resulting job also runs as
root.
6.1. Create the JAX Test Script (jax_test.py)
Save the following as jax_test.py:
# jax_test.py
# Simple MLP training loop in pure JAX to verify TPU execution.
import jax
import jax.numpy as jnp
import time
print("--- JAX TPU Model Training Verification ---")
print("JAX version:", jax.__version__)
print("Available devices:", jax.devices())
devices = jax.devices()
tpu_found = any(d.device_kind.lower().startswith('tpu') for d in devices)
if tpu_found:
print("SUCCESS: JAX is using TPU!")
else:
print("WARNING: JAX is NOT using TPU!")
# --- Simple MLP Training Loop ---
# Generate synthetic data
key = jax.random.PRNGKey(0)
key, x_key, y_key = jax.random.split(key, 3)
X = jax.random.normal(x_key, (1000, 64)) # 1000 samples, 64 features
# Target: simple non-linear function
Y = jnp.sin(X[:, 0:1]) + 0.5 * X[:, 1:2]
# Initialize weights (Simple 2-layer MLP: 64 -> 32 -> 1)
key, w1_key, w2_key = jax.random.split(key, 3)
w1 = jax.random.normal(w1_key, (64, 32)) * 0.1
b1 = jnp.zeros((32,))
w2 = jax.random.normal(w2_key, (32, 1)) * 0.1
b2 = jnp.zeros((1,))
params = {'w1': w1, 'b1': b1, 'w2': w2, 'b2': b2}
# Predict function
def predict(params, x):
h1 = jax.nn.relu(jnp.dot(x, params['w1']) + params['b1'])
return jnp.dot(h1, params['w2']) + params['b2']
# MSE Loss
def loss_fn(params, x, y):
preds = predict(params, x)
return jnp.mean((preds - y) ** 2)
# Update step (JIT-compiled for speed on TPU)
@jax.jit
def update_step(params, x, y, lr=0.05):
loss, grads = jax.value_and_grad(loss_fn)(params, x, y)
# SGD update using tree_map
new_params = jax.tree_util.tree_map(lambda p, g: p - lr * g, params, grads)
return new_params, loss
# Training Loop
print("Starting training loop (100 epochs)...")
start_time = time.time()
loss = 0.0
for epoch in range(100):
params, loss = update_step(params, X, Y)
if epoch % 10 == 0:
print(f"Epoch {epoch}: Loss = {loss:.6f}")
duration = time.time() - start_time
print(f"Training completed in {duration:.4f} seconds!")
print(f"Final Loss: {loss:.6f}")
print("-------------------------------------------")
6.2. Create the Slurm Submission Script (submit_jax.sh)
Save the following as submit_jax.sh. Note that it requests --gres=tpu:8 to
allocate all chips:
#!/usr/bin/env bash
#SBATCH --job-name=jax-tpu-test
#SBATCH --partition=tpu
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=tpu:8 # Request all 8 TPUs on the node
#SBATCH --output=jax_tpu_%j.out
#SBATCH --error=jax_tpu_%j.err
#SBATCH --time=00:15:00
set -euo pipefail
echo "=== Job started at $(date) ==="
echo "Running on node: $(hostname)"
# Install pip & venv dynamically (requires internet access in container)
echo "Installing system packages (pip and venv)..."
export DEBIAN_FRONTEND=noninteractive
apt-get update -y && apt-get install -y python3-pip python3-venv python3-full
# Create virtualenv and install JAX with TPU support
VENV_DIR="/tmp/jax_tpu_venv"
echo "Creating virtualenv..."
python3 -m venv --system-site-packages "$VENV_DIR"
source "$VENV_DIR"/bin/activate
echo "Installing JAX..."
python3 -m pip install --upgrade pip
python3 -m pip install "jax[tpu]" -f https://storage.googleapis.com/jax-releases/libtpu_releases.html
# Execute the JAX test
echo "Running JAX test script..."
python3 jax_test.py
echo "=== Job finished at $(date) ==="
6.3. Copy and Submit the Job
Copy the scripts to the worker pod and submit the job via Slurm:
# Copy scripts to the worker pod "slurm-tpu-worker-tpu-0"
kubectl cp jax_test.py slurm/slurm-tpu-worker-tpu-0:/tmp/jax_test.py -c slurmd
kubectl cp submit_jax.sh slurm/slurm-tpu-worker-tpu-0:/tmp/submit_jax.sh -c slurmd
# Submit the job
kubectl exec -n slurm slurm-tpu-worker-tpu-0 -c slurmd -- bash -c "cd /tmp && sbatch submit_jax.sh"
6.4. Verify Results
Once squeue shows the job has completed, check the output log
(/tmp/jax_tpu_<JOBID>.out):
kubectl exec -n slurm slurm-tpu-worker-tpu-0 -c slurmd -- cat /tmp/jax_tpu_1.out
It should look like the following, showing the successful JAX environment bootstrap, detection of the 8 TPU devices, and the loss successfully decreasing over the 100 training epochs:
=== Job started at Thu May 21 18:16:24 UTC 2026 ===
Running on node: tpu-0
Allocated GRES:
...
JAX successfully imported inside job!
Running JAX test script...
--- JAX TPU Model Training Verification ---
JAX version: 0.10.1
Available devices: [TpuDevice(id=0, process_index=0, coords=(0,0,0), core_on_chip=0), TpuDevice(id=1, process_index=0, coords=(1,0,0), core_on_chip=0), TpuDevice(id=2, process_index=0, coords=(0,1,0), core_on_chip=0), TpuDevice(id=3, process_index=0, coords=(1,1,0), core_on_chip=0), TpuDevice(id=4, process_index=0, coords=(0,2,0), core_on_chip=0), TpuDevice(id=5, process_index=0, coords=(1,2,0), core_on_chip=0), TpuDevice(id=6, process_index=0, coords=(0,3,0), core_on_chip=0), TpuDevice(id=7, process_index=0, coords=(1,3,0), core_on_chip=0)]
SUCCESS: JAX is using TPU!
Starting training loop (100 epochs)...
Epoch 0: Loss = 0.693804
Epoch 10: Loss = 0.535265
Epoch 20: Loss = 0.427468
Epoch 30: Loss = 0.329034
Epoch 40: Loss = 0.245817
Epoch 50: Loss = 0.187892
Epoch 60: Loss = 0.152073
Epoch 70: Loss = 0.129508
Epoch 80: Loss = 0.114930
Epoch 90: Loss = 0.104988
Training completed in 0.1254 seconds!
Final Loss: 0.098378
-------------------------------------------
=== Job finished at Thu May 21 18:16:38 UTC 2026 ===
This confirms a successful end-to-end TPU Slurm deployment on GKE!
7. Conclusion
This pattern enables you to quickly deploy Slurm clusters on GKE and run large-scale JAX training jobs leveraging Google Cloud’s physical TPU hardware slices!