TPU DRA

Slurm Bridge supports whole-host TPU requests through an indexed-gres DeviceProfile. Install the Google TPU DRA driver, enable DRAExtendedResource, and use a CDI-capable runtime. The driver needs tolerations for TPU and bridge-managed nodes.

Add this profile to the bridge Helm values, retaining any other profiles you use:

sharedConfig:
  deviceProfiles:
    - name: tpu.google.com
      driver: tpu.google.com
      selector: 'device.driver == "tpu.google.com"'
      backend:
        type: indexed-gres
        gresName: tpu

Add tpu to Slurm’s GresTypes. Hybrid nodes also need matching indexed gres.conf entries. The profile selector must exactly match the DeviceClass selector.

Apply the alias DeviceClass to request google.com/tpu through spec.extendedResourceName. The class matches the TPU profile above. This lets KubeRay detect TPUs and advertise Ray TPU resources, and lets LWS inject worker IDs and peer addresses. Use the RayJob or ordinary Job as a starting point. Ordinary Jobs and JobSets must supply TPU runtime coordination themselves.

Request all chips on a host in one container (four in the examples). Bridge creates the ResourceClaim. Partial-host requests fail in driver preparation; constrain placement to hosts with the requested chip count. Sharing is unsupported. Multi-host workloads also need a provisioned physical slice, Slurm placement constraints/topology, and worker coordination. Bridge does not provision slices, and these tests do not validate multi-host execution. The current homogeneous Ray/LWS gang reserves TPU GRES on every member’s node, including a CPU-only head or leader.

Kind check

The fixture runs the pinned upstream driver with four fake v4 devices per worker. Tests cover KubeRay/LWS detection, Slurm allocation, claim mappings, CDI device injection, cleanup, and reuse. They use one TPU worker plus a CPU head/leader; they do not run TPU computation.

export KUBECONFIG=/tmp/slinky-tpu.kubeconfig
export BUILDX_CONFIG=/tmp/slinky-tpu-buildx
export SKAFFOLD_CACHE_FILE=/tmp/slinky-tpu-skaffold-cache
export E2E_KUBE_CONTEXT=kind-slinky-tpu
./hack/kind.sh --core --dra-driver-google-tpu --kuberay slinky-tpu
E2E_LABELS=suite=scheduling E2E_RUN='^TestScheduling/TPU' make test-e2e
E2E_LABELS=suite=tpu E2E_RUN='^TestTPUAllocation$' make test-e2e
kind delete cluster --name slinky-tpu

For hybrid nodes, add --slurm-node-mode=hybrid to cluster creation and set SLURM_NODE_MODE=hybrid on the test commands. See Testing for cluster setup options. Fake devices and Kind values are for local tests only.