Administer Slurm for HPC workloads

Slurm support adds a persistent Slurm cluster to a Data Plane, with autoscaled worker capacity attached to it, so practitioners can submit batch HPC Jobs from a Workspace using the standard Slurm CLI. Slurm is enabled when the Data Plane is deployed, and cannot be added later. After that, your work is creating the two Compute Environments Slurm needs, sizing the worker Hardware Tiers, and making sure a shared Dataset is available on the Data Plane.

How it works

Domino runs High-Performance Computing (HPC) workloads on Slurm, deployed on Kubernetes with Slinky, SchedMD’s Slurm operator. Slurm is supported through the Distributed Compute Operator (DCO), like the other compute cluster types, but its deployment model differs in one important way: rather than one ephemeral cluster per execution, each Data Plane runs a single persistent Slurm cluster shared across users, with ephemeral, autoscaled compute attached to it. Slurm is a stateful scheduler, where priority, fair-share, backfill, and limits all depend on recent global state, so its central scheduling and accounting services persist on the Data Plane while worker capacity scales up and down with demand.

Slinky manages all Slurm cluster resources on Kubernetes, including Slurm’s central services: the controller, which schedules jobs, and the accounting service, which records the job runtime data used for quotas, access control, and Domino’s job monitoring.
Slurm is optional and enabled at deployment time. It adds services to the Data Plane that raise its baseline resource usage.
Slurm is part of the Data Plane. The Control Plane requires no additional components, so a configuration with Slurm absent from the Control Plane and the local Data Plane but installed on a remote Data Plane is valid. Domino enables the Slurm controls in the UI from the component information each Data Plane reports, and the Control Plane collects Slurm job statuses and metrics from the Data Plane no matter how a job was submitted.
Runtime statistics are stored permanently. The Control Plane captures the runtime statistics of every HPC Job and keeps them indefinitely, so the record outlives the job, the worker nodes that ran it, and the Workspace that submitted it.

Workspaces with an attached Slurm cluster act as Slurm login nodes. Domino injects the configuration and credentials the Slurm CLI needs.

Resources created at runtime

Nodesets: groups of worker nodes with identical characteristics, managed by the DCO. Each Workspace gets its own dedicated nodeset.
Partitions: logical scheduling queues, currently one per nodeset, managed by the Slurm controller through the nodeset custom resource.
Workers: compute nodes, scaled up and down automatically based on queued work. Each worker belongs to exactly one nodeset and one partition.

Execution flow

When a user launches a Workspace with an attached Slurm cluster:

The Control Plane builds the cluster manifest

The dispatcher creates a slurmcluster custom resource from the launch parameters and the Project configuration, and sends the manifest to the Data Plane.

The Data Plane Agent applies it

The agent applies the custom resource on the Data Plane cluster.

The Distributed Compute Operator provisions the cluster

The DCO creates a nodeset resource, from which the Slurm controller creates up to the user’s maximum number of worker pods, initially scaled to zero. It creates a token resource, from which the Slurm controller creates a JWT secret for the execution. It then adds the launching user to the slurm-users ConfigMap.

Prerequisites

A Domino deployment, on a local or remote Data Plane, on a supported provider.
Slurm enabled at installation time. Slurm cannot be added to an existing Data Plane without redeploying it.
A Dataset configured on the Slurm-enabled Data Plane. See Configure a shared Dataset.

Enable Slurm

Set compute_clusters.slurm.enabled to true in the installation configuration, or select Enable Slurm in your deployment tooling. For a remote Data Plane, also enable Data Plane storage and Data Plane Workspaces. No further Domino-side configuration is required for basic operation. Once Slurm is running, com.cerebro.domino.computegrid.computeCluster.slurm.minReplicaCount sets the minimum number of workers Domino keeps in each nodeset. It defaults to 0, so an idle cluster holds no worker capacity.

Create the Compute Environments

Slurm requires two Compute Environments: one with the Slurm CLI installed, which users select for the Workspace itself, and one that runs on the Slurm worker nodes. Create both by creating an Environment with the attributes below. Get the current image tags from the Compute Environment catalog.

Workspace Compute Environment

Environment Base: Start from a custom base image, then enter the Slurm Workspace image.
quay.io/domino/domino-slurm-environment:<tag>
Supported Clusters: None. This Environment runs the Workspace, not the workers.

Pluggable Workspace Tools: add the workspace tool definitions.

jupyter:
  title: "Jupyter (Python, R, Julia)"
  iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
  start: [ "/opt/domino/workspaces/jupyter/start" ]
  supportedFileExtensions: [ ".ipynb" ]
  httpProxy:
    port: 8888
    rewrite: false
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    requireSubdomain: false
jupyterlab:
  title: "JupyterLab"
  iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
  start: [ "/opt/domino/workspaces/jupyterlab/start" ]
  httpProxy:
    internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
    port: 8888
    rewrite: false
    requireSubdomain: false
vscode:
  title: "vscode"
  iconUrl: "/assets/images/workspace-logos/vscode.svg"
  start: [ "/opt/domino/workspaces/vscode/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false
rstudio:
  title: "RStudio"
  iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
  start: [ "/opt/domino/workspaces/rstudio/start" ]
  httpProxy:
    port: 8888
    requireSubdomain: false

Cluster Compute Environment

Environment Base: Start from a custom base image, then enter the Slurm cluster image.
quay.io/domino/slurm-cluster-environment:<tag>
Supported Clusters: Slurm.

The Slurm option in the Workspace launcher is enabled only when at least one Environment carrying the Slurm supported-cluster label is available to the user. Make both Environments visible to your intended users or their Organizations.

Create worker Hardware Tiers

Users select the worker tier that best fits their jobs when they attach a Slurm cluster to a Workspace. A typical deployment defines a small set of Hardware Tiers matched to common workload profiles:

Hardware Tier	Workload profile	Backing Kubernetes nodes
Slurm Compute	CPU-bound: simulations and parameter sweeps	Standard nodes with a high CPU-to-memory ratio
Slurm Big Memory	Memory-bound: large in-memory models and datasets	High-memory nodes
Slurm GPU	GPU-bound: training, GPU-accelerated simulation	GPU-enabled nodes

For predictable scheduling, size each Hardware Tier to match its underlying Kubernetes node. That avoids one Kubernetes node hosting several Slurm workers, and a Slurm worker sharing a node with unrelated workloads. When sizing, leave headroom for the small per-node overhead that infrastructure DaemonSets add. For a Kubernetes node with 8 CPUs and 32 GiB of memory, for example, a worker Hardware Tier of 6 CPUs and 28 GiB works well.

Configure a shared Dataset

Slurm has no internal mechanism for copying files between the Workspace and the worker nodes, so data is exchanged through a shared network file system. In Domino that is a Dataset, which makes configuring one on the Slurm-enabled Data Plane essential rather than optional.

Local Data Plane: configured by default at /domino/datasets/local/<project-name>.
Remote Data Plane: enable Data Plane storage and Workspaces on the Data Plane. See Manage Data Planes.

Set scheduling policies and limits

Configure Slurm’s supported scheduling policies and limits, including priority, fair-share, and per-user and per-Project limits on queued or running jobs, directly in Slurm with the standard Slurm administration tools. The entry point for CLI-based Slurm configuration is a Workspace on the Slurm-enabled Data Plane. The Slurm documentation covers the available policy and limit options.

Verify Slurm on a Data Plane

The Data Plane Agent reports installed component versions to the Control Plane, and the Slurm controller is one of them. To confirm that a Data Plane supports Slurm, find the status message from that Data Plane in the data plane service log on the Control Plane:

Data Plane Local (...) current status: {"id":"...","state":"Healthy",...,"componentVersions":{"slurm/slurmctld":"25.11.3-r3", ...}}

The Data Plane state must be Healthy, and componentVersions must include a slurm/slurmctld entry. The version itself does not matter.

Enable HPC Job submission from the Jobs page

The HPC tab on a Project’s Jobs page carries the history of the Project’s HPC Jobs, and lists them no matter how they were submitted. Submission from that tab is gated behind the ShortLived.EnableHPCJobs feature flag. Set it to true and a Run button appears on the tab, letting users submit HPC Jobs from the Domino UI and the API rather than from a Workspace terminal. Domino ships the gate because the submission experience is still being refined. Submission from a Workspace terminal is unaffected and needs no flag.

Map Domino users to Slurm users

Each Slurm user name derives from the name of the Domino user who launched the Workspace, so Slurm accounting, fair-share, and per-user limits apply per person rather than to one shared account. A Domino user name has to be a valid Linux user name for this mapping to work: a lowercase letter or underscore followed by up to 31 lowercase letters, digits, underscores, or hyphens, matching ^[a-z_][a-z0-9_-]{0,31}. It must also not be a reserved system name such as root, daemon, or nobody.

Launching a Workspace with an attached Slurm cluster fails when the Domino user name breaks either rule. Check user names against both before you make Slurm available to a group of users.

A Domino user holding the SysAdmin role maps instead to Slurm’s built-in slurm user, which is a Slurm system administrator.

Limitations

A Slurm cluster is attached at Workspace launch, and the Workspace is the only place the Slurm CLI runs. A standard Domino Job cannot carry an attached Slurm cluster.
Nodesets and partitions are created per Workspace, so each Workspace gets its own dedicated nodeset.
Data is exchanged between the Workspace and the workers only through Datasets, over NFS. Workers cannot reach Data Sources or NetApp Volumes.
Integration with external, customer-managed Slurm clusters is not supported.
Slurm cannot be added to a Data Plane that is already deployed.

Jobs on worker nodes are not sandboxed from one another. Jobs from different users can run on the same worker node, and all Domino users share a single Unix group there, so files written with group-readable or group-writable permissions are visible to other users. Tell users to set a restrictive umask on sensitive output files.

Known issues

Slurm REST API memory growth

Broad sacct queries, such as sacct -a, run after high job volume cause sustained memory growth in the Slurm REST API. Memory is not promptly released, which can lead to out-of-memory restarts of the service. To mitigate it:

Tell users to filter sacct queries by job ID, user, or time range rather than querying all jobs.
Monitor pod memory with kubectl top pod -n domino-compute.
Restart the service when needed with kubectl rollout restart deployment slurm-restapi -n domino-compute.

Troubleshooting

The Slurm option is disabled in the Workspace launcher

Two conditions must both hold: at least one Environment labeled with the Slurm supported-cluster type is available to the user, and the Data Plane they selected has Slurm installed.

The Slurm option is disabled when launching a Job

Standard Jobs do not support attached Slurm clusters. Users submit HPC Jobs from a Workspace terminal, or from the HPC tab of the Project’s Jobs page once you turn on the flag described in Enable HPC Job submission from the Jobs page.

Jobs never dispatch

Verify that worker pods are being created with kubectl get pods -n domino-compute, and that the requested resources fit within the worker Hardware Tier and the maximum number of workers set on the cluster.

Run HPC workloads with Slurm: the practitioner-facing guide to submitting HPC Jobs.
Manage Hardware Tiers: create and size the worker tiers.
Manage Data Planes: register a Data Plane and enable it for storage and Workspaces.
Datasets: the shared storage Slurm jobs read and write.
Central configuration: the Slurm worker configuration keys.

Architecture & deployment

Domino installation

Identity & access

Infrastructure & compute

Platform configuration

Data administration

Operations, monitoring & cost

Security & compliance

Troubleshooting

Administer Slurm for HPC workloads

How it works

Resources created at runtime

Execution flow

Prerequisites

Enable Slurm

Create the Compute Environments

Workspace Compute Environment

Cluster Compute Environment

Create worker Hardware Tiers

Configure a shared Dataset

Set scheduling policies and limits

Verify Slurm on a Data Plane

Enable HPC Job submission from the Jobs page

Map Domino users to Slurm users

Limitations

Known issues

Slurm REST API memory growth

Troubleshooting

​How it works

​Resources created at runtime

​Execution flow

​Prerequisites

​Enable Slurm

​Create the Compute Environments

​Workspace Compute Environment

​Cluster Compute Environment

​Create worker Hardware Tiers

​Configure a shared Dataset

​Set scheduling policies and limits

​Verify Slurm on a Data Plane

​Enable HPC Job submission from the Jobs page

​Map Domino users to Slurm users

​Limitations

​Known issues

​Slurm REST API memory growth

​Troubleshooting

​Related

How it works

Resources created at runtime

Execution flow

Prerequisites

Enable Slurm

Create the Compute Environments

Workspace Compute Environment

Cluster Compute Environment

Create worker Hardware Tiers

Configure a shared Dataset

Set scheduling policies and limits

Verify Slurm on a Data Plane

Enable HPC Job submission from the Jobs page

Map Domino users to Slurm users

Limitations

Known issues

Slurm REST API memory growth

Troubleshooting

Related