Run HPC workloads with Slurm

Domino runs High-Performance Computing (HPC) workloads on Slurm, the standard workload manager in HPC. Submit your existing Slurm job scripts from inside a Domino Workspace with the standard sbatch, squeue, and scancel commands, without rewriting the scripts or changing how you work. Job scripts driven by common R and Python frameworks such as mirai and clustermq run unchanged.

How it works

Slurm is a cluster resource manager and job scheduler: it allocates nodes, CPUs, GPUs, and memory to jobs and runs arbitrary executables, with no knowledge of the computation inside. Reach for a Slurm cluster when you have batch-oriented HPC workloads, such as simulations, statistical ensembles, or parallel parameter sweeps, that are already written as Slurm job scripts or that fit the batch scheduling model. When you launch a Workspace with an attached Slurm cluster, the Workspace acts as your Slurm login node. Domino injects the configuration and credentials the Slurm CLI needs, and you submit jobs from the Workspace terminal exactly as you would on a traditional HPC cluster. The HPC tab on the Project’s Jobs page is the second path: it submits a script that lives in the Project, without the Slurm CLI, once an administrator enables it. See Submit HPC Jobs from the Jobs page.

Key terms

Term	Definition
HPC Job	A batch job submitted to and scheduled by Slurm from inside Domino.
Job array	A set of related jobs submitted as one unit, indexed by an array task ID, such as a parameter sweep.
Partition	A Slurm scheduling queue: a logical grouping of worker nodes.
Worker	A compute node in the Slurm cluster. Domino autoscales workers up and down based on queued work.

Prerequisites

Your administrator has deployed Domino with Slurm support enabled on at least one Data Plane. See Administer Slurm for HPC workloads.
Two Compute Environments are available to you: one carrying the Slurm CLI, which runs the Workspace, and one that runs on the Slurm worker nodes. Your administrator creates both.
A worker Hardware Tier sized for Slurm workloads is available to you. Your administrator defines one tier per common workload profile, such as CPU-bound, memory-bound, and GPU-bound work, and you pick the one that fits your jobs when you attach the cluster.
A Dataset is configured on the Slurm-enabled Data Plane. Slurm copies no files between the Workspace and the worker nodes, so a shared Dataset is the common file system for job scripts, inputs, and outputs.

Attach a Slurm cluster to a Workspace

Select the Slurm Workspace Environment

In the Workspace launcher, on the Environment tab, select the Slurm-capable Workspace Environment your administrator created.

Select a Slurm-enabled Data Plane

On the Data Plane & Hardware tab, select a Hardware Tier associated with a Data Plane that has Slurm installed. Confirm that a Dataset is listed under Datasets available.

Attach the cluster

On the Compute Cluster tab, select Slurm, then set the Worker Hardware Tier for the Slurm worker nodes, the Max Number of Workers that Domino scales the partition up to based on queued jobs, and the Cluster Compute Environment designated for Slurm.

Launch the Workspace

The Slurm CLI is available in the Workspace terminal once the Workspace starts.

If the Slurm option is disabled, see Troubleshooting.

Verify the cluster

Open a terminal in your Workspace and run sinfo to list the available partitions:

$ sinfo
PARTITION                      AVAIL  TIMELIMIT  NODES  STATE NODELIST
slurm-69f2150f95cca76755bcf00e    up   infinite      0    n/a
all*                              up   infinite      0    n/a

Zero nodes is normal for an idle cluster. Domino provisions workers on demand when jobs enter the queue.

Slurm commands

All standard Slurm commands are available from the Workspace terminal:

Command	Purpose	Example
`sinfo`	Show partitions and node availability	`sinfo`
`sbatch`	Submit a batch job script	`sbatch job.sh`
`squeue`	Show queued and running jobs	`squeue -j <jobid>`
`scontrol`	Show full details of a queued or running job	`scontrol show job <jobid>`
`sacct`	Show accounting data for completed jobs	`sacct -j <jobid>`
`scancel`	Cancel a queued or running job	`scancel <jobid>`

Give resource requests either as sbatch command-line options or as #SBATCH directives in the script header. Command-line options override header directives:

sbatch --job-name=hello_job --time=00:01:00 --cpus-per-task=1 --mem=256M job.sh

Prefer specific sacct queries, filtered by job ID, user, or time range, over broad ones such as sacct -a. Broad queries after high job volume drive memory growth in the Slurm REST API.

Submit your first HPC Job

Slurm workers share files with your Workspace through the Dataset, so job scripts must write output to the Dataset path rather than to Workspace storage. The example below uses /domino/datasets/local/quick-start. Adjust it to match the Dataset configured in your Project and Data Plane.

Create the job script

In your Workspace, create sample-job.sh in the default working directory with the following contents. No permission changes are needed. The job runs three array tasks, each printing diagnostic information about the worker node it ran on. It redirects its own output to the Dataset, because Domino controls --output and --error and overwrites whatever you set them to.

#!/bin/bash
#SBATCH --job-name=hello-cluster
#SBATCH --array=1-3
#SBATCH --nodes=1
#SBATCH --ntasks=1

OUTDIR="/domino/datasets/local/quick-start"
OUTFILE="${OUTDIR}/output_task_${SLURM_ARRAY_TASK_ID}.txt"

{
  echo "===== Task Info ====="
  echo "Job ID:          ${SLURM_JOB_ID}"
  echo "Array Job ID:    ${SLURM_ARRAY_JOB_ID}"
  echo "Array Task ID:   ${SLURM_ARRAY_TASK_ID}"
  echo "Running on node: $(hostname)"
  echo "Started at:      $(date -u '+%Y-%m-%dT%H:%M:%SZ')"

  echo ""
  echo "===== CPU Info ====="
  nproc
  lscpu | grep -E '^(CPU\(s\)|Model name|Architecture)'

  echo ""
  echo "===== Memory Info ====="
  free -h

  echo ""
  echo "===== Shared Storage Check ====="
  df -h "${OUTDIR}"

  echo ""
  echo "===== Simple Compute ====="
  python3 -c "
import time, math
start = time.time()
result = sum(math.sqrt(i) for i in range(1, 10_000_000))
elapsed = time.time() - start
print(f'Sum of sqrts: {result:.4f}')
print(f'Elapsed:      {elapsed:.3f}s')
"

  echo ""
  echo "Finished at: $(date -u '+%Y-%m-%dT%H:%M:%SZ')"
} > "${OUTFILE}" 2>&1

echo "Task ${SLURM_ARRAY_TASK_ID} wrote output to ${OUTFILE}"

exit 0

Submit the job

Submit the script with sbatch. Slurm returns the job ID it assigned:

$ sbatch sample-job.sh
Submitted batch job 1

Check the job status

Check the queue immediately with squeue:

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           1_[1-3] slurm-69f hello-cl     1000 PD       0:00      1 (PartitionConfig)

State PD means the job is pending while Domino provisions workers, which is normal on an idle cluster.

Watch the lifecycle

While the job is queued and running:

sinfo shows worker nodes provisioned as the partition scales up.
squeue shows the job move from pending (PD) to running (R).
scontrol show job <jobid> gives full details while the job is queued or running.
sacct -j <jobid> gives the accounting record once the job completes.

This job finishes in seconds, so it can complete before you see it running. To watch the lifecycle unfold, add sleep 180 at the end of the script. You can also submit the job several times: jobs queue and are dispatched as workers become available.

Read the output

When squeue returns empty, the job has finished. Check the output files in your shared Dataset directory:

ls /domino/datasets/local/quick-start/
cat /domino/datasets/local/quick-start/output_task_1.txt

Stop the Workspace

The cluster’s dedicated worker nodeset is tied to the Workspace, so stopping the Workspace releases the compute resources.

Submit HPC Jobs from the Jobs page

The HPC tab on a Project’s Jobs page carries the history of the Project’s HPC Jobs, with each job’s status and metrics, no matter how the job was submitted. A Run button on that tab submits an HPC Job without the Slurm CLI. The script has to live in the Project, and you give its path relative to the Project root rather than a Dataset path, which is the one place this differs from submitting through sbatch. Alongside the script you pick the Data Plane to run on and, optionally, the CPU, GPU, memory, node, task, and wall-time requests. The Domino API exposes the same operation as POST /api/hpc/v1/jobs, with companion endpoints to list, inspect, cancel, and fetch logs for HPC Jobs.

The Run button appears only where an administrator has enabled the ShortLived.EnableHPCJobs feature flag. Domino gates UI and API submission behind it while the submission experience is refined. Submitting from a Workspace terminal needs no flag.

Supported HPC Job types

Job type	Example	Notes
Serial shell scripts	`sbatch job.sh`	Resource requests through command-line options or the `#SBATCH` header.
SMP (single-node, multi-core)	`#SBATCH --cpus-per-task=4`, `#SBATCH --mem=1G`	`--cpus-per-task` sets cores per task, `--mem` sets memory per node.
Array jobs	`#SBATCH --array=0-9`	Use `$SLURM_ARRAY_JOB_ID` and `$SLURM_ARRAY_TASK_ID` in the script to give each task its own output path.
Dependent jobs	`sbatch --dependency=afterok:<jobid> job.sh`	Run a job only after another completes, with `afterok`, `afterany`, or `afternotok`.

Monitor and manage HPC Jobs

Track jobs from the Workspace terminal with squeue, scontrol show job <jobid>, and sacct, and cancel them with scancel <jobid>. The Control Plane captures the runtime statistics of every HPC Job and stores them permanently, so the record outlives the job, the worker nodes that ran it, and the Workspace that submitted it. Inside your job scripts, express resource requests in Slurm-native terms: CPUs, GPUs, memory, and wall time. Slurm schedules each job onto workers of the Worker Hardware Tier you selected for the cluster, and Domino autoscales those workers up and down based on the queue.

Limitations

A Slurm cluster is attached at Workspace launch, and the Workspace is the only place the Slurm CLI runs. A standard Domino Job cannot carry an attached Slurm cluster.
Data moves between the Workspace and the workers only through Datasets, a shared network file system. Workers cannot reach Data Sources or NetApp Volumes.
Interactive and scheduled Slurm jobs are not supported.

Jobs from different users can run on the same worker node and are not sandboxed from one another. All Domino users share a single Unix group on the workers, so files written with group-readable or group-writable permissions are visible to other users. Set a restrictive umask on sensitive output files.

Troubleshooting

The Slurm option is disabled in the Workspace launcher

Two conditions must both hold: at least one Environment labeled with the Slurm supported-cluster type is available to you, and the Data Plane selected on the Data Plane & Hardware tab has Slurm installed. Check your Environment selection and Data Plane, or ask your administrator.

Job output files are missing

Domino overwrites --output and --error on every HPC Job so that it can collect job output into the system of record, so both resolve to predefined locations and setting them in your script has no effect. To keep your own copy of a job’s output, redirect it inside the script to the shared Dataset directory, as the sample job script does. Worker nodes cannot write to your Workspace file system, so any path outside the Dataset fails.

Jobs stay pending longer than expected

Domino autoscales workers on demand, so a cold cluster needs time to provision nodes. If jobs never dispatch, check that your resource request fits within the Worker Hardware Tier and the configured maximum number of workers, or ask your administrator about scheduling limits.

Administer Slurm for HPC workloads: enable Slurm, create the Environments, and size the worker Hardware Tiers.
On-demand distributed computing: the other cluster types Domino provisions.
Distributed GPUs with Open MPI: run MPI programs on a Domino-managed MPI cluster.
Datasets: the shared storage Slurm jobs read and write.

Core Concepts

Features

Run HPC workloads with Slurm

How it works

Key terms

Prerequisites

Attach a Slurm cluster to a Workspace

Verify the cluster

Slurm commands

Submit your first HPC Job

Submit HPC Jobs from the Jobs page

Supported HPC Job types

Monitor and manage HPC Jobs

Limitations

Troubleshooting

​How it works

​Key terms

​Prerequisites

​Attach a Slurm cluster to a Workspace

​Verify the cluster

​Slurm commands

​Submit your first HPC Job

​Submit HPC Jobs from the Jobs page

​Supported HPC Job types

​Monitor and manage HPC Jobs

​Limitations

​Troubleshooting

​Related

How it works

Key terms

Prerequisites

Attach a Slurm cluster to a Workspace

Verify the cluster

Slurm commands

Submit your first HPC Job

Submit HPC Jobs from the Jobs page

Supported HPC Job types

Monitor and manage HPC Jobs

Limitations

Troubleshooting

Related