Admission
Table of Contents
Overview
The Kubernetes documentation defines admission controllers as:
a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the resource, but after the request is authenticated and authorized.
It also states that:
Admission control mechanisms may be validating, mutating, or both. Mutating controllers may modify the data for the resource being modified; validating controllers may not.
The slurm-bridge admission controller is both mutating and validating. It
modifies pods within namespaces specified in helm/slurm-bridge/values.yaml to
use the slurm-bridge scheduler instead of the default Kubernetes scheduler,
and rejects unsupported scheduling and resource configurations.
Design
Pods created in the specified namespaces with the default Kubernetes scheduler
have .spec.schedulerName changed to the slurm-bridge scheduler. Explicitly
selected custom schedulers are preserved. Pods selecting the bridge scheduler,
including those outside managed namespaces, receive its managed-node toleration
and, for unassigned pods that require registration, a scheduling gate. These
pods remain SchedulingGated until the controller registers their Slurm
allocation. The Helm webhooks exclude kube-system and the bridge’s
installation namespace.
Managed namespaces are defined as a list of namespaces as configured in the
admission controller’s values.yaml for managedNamespaces[]. Alternatively, a
managedNamespaceSelector can be used to select namespaces based on labels. If
managedNamespaceSelector is set, managedNamespaces will be ignored.
Pods selected for the slurm-bridge scheduler receive the
slinky.slurm.net/managed-node toleration during admission. If admission is
disabled or the pod’s namespace is excluded from the webhook, set this
toleration explicitly to run on bridged nodes. The scheduler treats the pods it
receives as immutable and does not add tolerations.
Managed pods can request either native cpu or the CPU DRA extended resource
deviceclass.resource.kubernetes.io/dra.cpu, but cannot specify both. CPU DRA
must be requested explicitly; native cpu requests are not converted to DRA
requests by the admission controller.
CPU DRA requires slurmjob.slinky.slurm.net/exclusive: "false" on the
workload’s annotation source. Standalone pods that
omit it or enable exclusive allocation are rejected at admission. For
controller-owned and grouped pods, the scheduler checks the resolved annotations
before releasing the held Slurm job for allocation. Native CPU and GPU-only
requests still support exclusive allocation.
In-place resizing is not supported for managed pods. The admission controller
rejects requests to the pods/resize subresource so Kubernetes resources cannot
diverge from the corresponding Slurm allocation.
Pods cannot provide spec.resourceClaims. Request DRA devices through
DeviceClass extended resources instead.
The slurmjob.slinky.slurm.net/constraints annotation is combined with the
slurm_bridge_gres_compatible node feature that every bridge job requires. The
admission controller rejects Pod annotations containing constraint expressions
that Slurm’s grammar does not allow to be combined with an additional feature,
such as a bare feature count (rack1*2) or an OR outside parentheses mixed with
parentheses or brackets ((a&b)|(c&d)), so the job fails at creation rather
than on every scheduling cycle. Annotations on owning workloads and PodGroups
are resolved later and checked before Slurm submission, not by the Pod admission
webhook. See supported Slurm job annotations for the supported forms.
Managed pods are also validated against the supported DRA DeviceClass set.
Unsupported DeviceClass resources in requests or limits are rejected for init
containers and regular containers. Operators can extend the supported set with
configured device profiles. See Device resources for the built-in classes,
profile configuration, and legacy device-plugin resources. Pods outside managed
namespaces that do not select the slurm-bridge scheduler are not subject to
this validation.
Sequence Diagram
Workload annotation deprecation
By default, a ValidatingAdmissionPolicy warns about top-level
slurmjob.slinky.slurm.net/* annotations on Jobs, JobSets, LeaderWorkerSets,
PodGroups, CompositePodGroups, and Workloads when they are created. Requests
remain allowed. Move annotations to Pod templates;
Pod annotations take precedence during the
transition. To reject these annotations instead, install or upgrade with
--set admission.workloadAnnotationAction=Deny. Set it to Warn (the default)
to allow requests with a warning. The policy only checks CREATE requests in both
modes; updates to existing workloads remain allowed.
The planned transition is warnings, then denial by default, then removal of legacy annotation support. Release timing will depend on deprecation feedback.
The policy uses admission.managedNamespaces or
admission.managedNamespaceSelector, excluding kube-system and the bridge
installation namespace. Pod and Pod template annotations are allowed in both
modes. Disabling admission or clearing both namespace settings omits the policy.
Unsupported scheduling requests
When admission is enabled, a ValidatingAdmissionPolicy checks pods on creation
after scheduler defaulting. It applies to pods selecting the bridge scheduler
explicitly or through a managed namespace. It excludes kube-system and the
bridge installation namespace.
On creation, the mutating webhook removes an explicit spec.nodeName from
bridge pods and returns a warning so Slurm can select the node. This applies in
both strict and warning modes; updates retain their existing node assignment.
The policy rejects these requests by default:
A nonempty
spec.nodeSelector, required or preferred node affinity, pod affinity or anti-affinity, and topology spread constraints. Use a Slurm partition or constraint for node placement.A volume source outside the supported set of
emptyDir,configMap,secret,projected,downwardAPI,image,nfs, andpersistentVolumeClaim. This includeshostPath, inline CSI, and generic ephemeral volumes. PVCs still go through Kubernetes VolumeBinding checks.
Set --set admission.strictPodValidation=false to admit these pods with a
warning. Slurm may place a pod without honoring the requested constraints. Pod
updates, including binding and registration, are outside this policy.