Pyxis

Overview

This guide tells how to configure your Slurm cluster to use pyxis (and enroot), a Slurm SPANK plugin for containerized jobs with Nvidia GPU support.

Configure

Configure plugstack.conf to include the pyxis configuration.

Warning: In plugstack.conf, you must use glob syntax to avoid slurmctld failure while trying to resolve the paths in the includes. Only the login and slurmd pods should actually have the pyxis libraries installed.

slurm:
  configFiles:
    plugstack.conf: |
      include /usr/share/pyxis/*
  ...

Configure one or more NodeSets and the login pods to use a pyxis OCI image.

login:
  image:
    repository: ghcr.io/slinkyproject/login-pyxis
  ...
compute:
  nodesets:
    - name: debug
      image:
        repository: ghcr.io/slinkyproject/slurmd-pyxis
      ...

To make enroot activity in the login container permissible, it requires securityContext.privileged=true.

login:
  image:
    repository: ghcr.io/slinkyproject/login-pyxis
  securityContext:
    privileged: true

Test

Submit a job to a Slurm node.

$ srun --partition=debug grep PRETTY /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"

Submit a job to a Slurm node with pyxis and it will launch in its requested container.

$ srun --partition=debug --container-image=alpine:latest grep PRETTY /etc/os-release
pyxis: importing docker image: alpine:latest
pyxis: imported docker image: alpine:latest
PRETTY_NAME="Alpine Linux v3.21"

Warning: SPANK plugins will only work on specific Slurm node that have them and is configured to use them. It is best to constrain where jobs run with --partition=<partition>, --batch=<features>, and/or --constraint=<features> to ensure a compatible computing environment.

If the login container has securityContext.privileged=true, enroot activity is permissible. You can test the functionality with the following:

enroot import docker://alpine:latest