RestAPI Job Submission Guide

Table of Contents

Overview

This guide provides an overview of the configuration required to submit jobs to the Slurm RestAPI when running in Slurm-operator. For complete documentation on the Slurm RestAPI, please refer to the official documentation.

Container Configuration

To generate a JWT with scontrol token, the user must be resolvable in the container where that command runs. For this example, that is the LoginSet pod. The generated Slurm configuration sets AuthInfo=use_client_ids, so the same user does not also need to exist in the slurmctld container. The compute environment must still be able to resolve the submitted user’s UID and GID when the job runs.

There are several ways that a user can be made resolvable in the login and compute environments. In a production environment, the installation and configuration of an identity provider through SSSD is the recommended deployment pattern.

In a development environment, the user and group may instead be created in the Dockerfiles used to build the login and slurmd container images.

For more information on building Slinky containers, please see containers. Configuration files such as sssd.conf can be supplied through the Slurm chart or mounted into pod containers as described on this page.

Job Submission Process

Creating a JWT

To submit a job using the Slurm RestAPI, a user must first generate a JWT using the scontrol token command. To do this, the user should ssh to the deployed LoginSet and execute the scontrol token command.

Warning

This token is all that is required in order to submit jobs as a user. As such, it should be protected like a password!

Once the user has generated a JWT, they should export that token as an environment variable in their shell. For the purposes of this demo, we will assume that this environment variable is $SLURM_JWT.

Submitting a Job

Once a token has been generated and exported as an environment variable, a user can submit jobs via the Slurm RestAPI. Below is a simple job that executes srun hostname on a single node in the all partition:

curl -sS -o /tmp/curl.log -k -vvvv \
  -H "X-SLURM-USER-TOKEN: $SLURM_JWT" \
  -H "Content-Type: application/json" \
  -X POST "http://172.18.0.8:6820/slurm/v0.0.45/job/submit" \
  -d '{
    "job": {
      "name": "demo",
      "partition": "all",
      "current_working_directory": "/tmp",
      "environment": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
      ],
      "nodes": "1",
      "tasks": 1
    },
    "script": "#!/bin/bash\nsrun hostname\n"
  }'

After submitting the job via the RestAPI, you should see a response with the status HTTP/1.1 200 OK, indicating that the job was submitted.