> ## Documentation Index > Fetch the complete documentation index at: https://docs.domino.ai/llms.txt > Use this file to discover all available pages before exploring further. # Run HPC workloads with Slurm > Run existing Slurm batch scripts on Domino, from a Workspace or the Jobs page, on worker nodes Domino autoscales. Domino runs High-Performance Computing (HPC) workloads on [Slurm](https://slurm.schedmd.com/), the standard workload manager in HPC. Submit your existing Slurm job scripts from inside a Domino Workspace with the standard `sbatch`, `squeue`, and `scancel` commands, without rewriting the scripts or changing how you work. Job scripts driven by common R and Python frameworks such as `mirai` and `clustermq` run unchanged. ## How it works Slurm is a cluster resource manager and job scheduler: it allocates nodes, CPUs, GPUs, and memory to jobs and runs arbitrary executables, with no knowledge of the computation inside. Reach for a Slurm cluster when you have batch-oriented HPC workloads, such as simulations, statistical ensembles, or parallel parameter sweeps, that are already written as Slurm job scripts or that fit the batch scheduling model. When you launch a Workspace with an attached Slurm cluster, the Workspace acts as your Slurm login node. Domino injects the configuration and credentials the Slurm CLI needs, and you submit jobs from the Workspace terminal exactly as you would on a traditional HPC cluster. The **HPC** tab on the Project's Jobs page is the second path: it submits a script that lives in the Project, without the Slurm CLI, once an administrator enables it. See [Submit HPC Jobs from the Jobs page](#submit-hpc-jobs-from-the-jobs-page). ### Key terms | Term | Definition | | :-------- | :--------------------------------------------------------------------------------------------------- | | HPC Job | A batch job submitted to and scheduled by Slurm from inside Domino. | | Job array | A set of related jobs submitted as one unit, indexed by an array task ID, such as a parameter sweep. | | Partition | A Slurm scheduling queue: a logical grouping of worker nodes. | | Worker | A compute node in the Slurm cluster. Domino autoscales workers up and down based on queued work. | ## Prerequisites * Your administrator has deployed Domino with Slurm support enabled on at least one [Data Plane](/6.3/platform-capabilities/core-concepts/data-plane). See [Administer Slurm for HPC workloads](/6.3/admin/infrastructure-and-compute/administer-slurm). * Two Compute Environments are available to you: one carrying the Slurm CLI, which runs the Workspace, and one that runs on the Slurm worker nodes. Your administrator creates both. * A worker Hardware Tier sized for Slurm workloads is available to you. Your administrator defines one tier per common workload profile, such as CPU-bound, memory-bound, and GPU-bound work, and you pick the one that fits your jobs when you attach the cluster. * A [Dataset](/6.3/platform-capabilities/core-concepts/data/datasets) is configured on the Slurm-enabled Data Plane. Slurm copies no files between the Workspace and the worker nodes, so a shared Dataset is the common file system for job scripts, inputs, and outputs. ## Attach a Slurm cluster to a Workspace In the [Workspace launcher](/6.3/platform-capabilities/core-concepts/workspaces/launch-a-workspace), on the **Environment** tab, select the Slurm-capable Workspace Environment your administrator created. On the **Data Plane & Hardware** tab, select a Hardware Tier associated with a Data Plane that has Slurm installed. Confirm that a Dataset is listed under **Datasets available**. On the **Compute Cluster** tab, select **Slurm**, then set the **Worker Hardware Tier** for the Slurm worker nodes, the **Max Number of Workers** that Domino scales the partition up to based on queued jobs, and the **Cluster Compute Environment** designated for Slurm. The Slurm CLI is available in the Workspace terminal once the Workspace starts. If the **Slurm** option is disabled, see [Troubleshooting](#troubleshooting). ## Verify the cluster Open a terminal in your Workspace and run `sinfo` to list the available partitions: ```bash theme={null} $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST slurm-69f2150f95cca76755bcf00e up infinite 0 n/a all* up infinite 0 n/a ``` Zero nodes is normal for an idle cluster. Domino provisions workers on demand when jobs enter the queue. ## Slurm commands All standard Slurm commands are available from the Workspace terminal: | Command | Purpose | Example | | :--------- | :------------------------------------------- | :-------------------------- | | `sinfo` | Show partitions and node availability | `sinfo` | | `sbatch` | Submit a batch job script | `sbatch job.sh` | | `squeue` | Show queued and running jobs | `squeue -j ` | | `scontrol` | Show full details of a queued or running job | `scontrol show job ` | | `sacct` | Show accounting data for completed jobs | `sacct -j ` | | `scancel` | Cancel a queued or running job | `scancel ` | Give resource requests either as `sbatch` command-line options or as `#SBATCH` directives in the script header. Command-line options override header directives: ```bash theme={null} sbatch --job-name=hello_job --time=00:01:00 --cpus-per-task=1 --mem=256M job.sh ``` Prefer specific `sacct` queries, filtered by job ID, user, or time range, over broad ones such as `sacct -a`. Broad queries after high job volume drive memory growth in the Slurm REST API. ## Submit your first HPC Job Slurm workers share files with your Workspace through the Dataset, so job scripts must write output to the Dataset path rather than to Workspace storage. The example below uses `/domino/datasets/local/quick-start`. Adjust it to match the Dataset configured in your Project and Data Plane. In your Workspace, create `sample-job.sh` in the default working directory with the following contents. No permission changes are needed. The job runs three array tasks, each printing diagnostic information about the worker node it ran on. It redirects its own output to the Dataset, because Domino controls `--output` and `--error` and overwrites whatever you set them to. ```bash theme={null} #!/bin/bash #SBATCH --job-name=hello-cluster #SBATCH --array=1-3 #SBATCH --nodes=1 #SBATCH --ntasks=1 OUTDIR="/domino/datasets/local/quick-start" OUTFILE="${OUTDIR}/output_task_${SLURM_ARRAY_TASK_ID}.txt" { echo "===== Task Info =====" echo "Job ID: ${SLURM_JOB_ID}" echo "Array Job ID: ${SLURM_ARRAY_JOB_ID}" echo "Array Task ID: ${SLURM_ARRAY_TASK_ID}" echo "Running on node: $(hostname)" echo "Started at: $(date -u '+%Y-%m-%dT%H:%M:%SZ')" echo "" echo "===== CPU Info =====" nproc lscpu | grep -E '^(CPU\(s\)|Model name|Architecture)' echo "" echo "===== Memory Info =====" free -h echo "" echo "===== Shared Storage Check =====" df -h "${OUTDIR}" echo "" echo "===== Simple Compute =====" python3 -c " import time, math start = time.time() result = sum(math.sqrt(i) for i in range(1, 10_000_000)) elapsed = time.time() - start print(f'Sum of sqrts: {result:.4f}') print(f'Elapsed: {elapsed:.3f}s') " echo "" echo "Finished at: $(date -u '+%Y-%m-%dT%H:%M:%SZ')" } > "${OUTFILE}" 2>&1 echo "Task ${SLURM_ARRAY_TASK_ID} wrote output to ${OUTFILE}" exit 0 ``` Submit the script with `sbatch`. Slurm returns the job ID it assigned: ```bash theme={null} $ sbatch sample-job.sh Submitted batch job 1 ``` Check the queue immediately with `squeue`: ```bash theme={null} $ squeue JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 1_[1-3] slurm-69f hello-cl 1000 PD 0:00 1 (PartitionConfig) ``` State `PD` means the job is pending while Domino provisions workers, which is normal on an idle cluster. While the job is queued and running: * `sinfo` shows worker nodes provisioned as the partition scales up. * `squeue` shows the job move from pending (`PD`) to running (`R`). * `scontrol show job ` gives full details while the job is queued or running. * `sacct -j ` gives the accounting record once the job completes. This job finishes in seconds, so it can complete before you see it running. To watch the lifecycle unfold, add `sleep 180` at the end of the script. You can also submit the job several times: jobs queue and are dispatched as workers become available. When `squeue` returns empty, the job has finished. Check the output files in your shared Dataset directory: ```bash theme={null} ls /domino/datasets/local/quick-start/ cat /domino/datasets/local/quick-start/output_task_1.txt ``` The cluster's dedicated worker nodeset is tied to the Workspace, so stopping the Workspace releases the compute resources. ## Submit HPC Jobs from the Jobs page The **HPC** tab on a Project's [Jobs](/6.3/platform-capabilities/core-concepts/jobs) page carries the history of the Project's HPC Jobs, with each job's status and metrics, no matter how the job was submitted. A **Run** button on that tab submits an HPC Job without the Slurm CLI. The script has to live in the Project, and you give its path relative to the Project root rather than a Dataset path, which is the one place this differs from submitting through `sbatch`. Alongside the script you pick the Data Plane to run on and, optionally, the CPU, GPU, memory, node, task, and wall-time requests. The [Domino API](/6.3/reference/api/domino-open-api) exposes the same operation as `POST /api/hpc/v1/jobs`, with companion endpoints to list, inspect, cancel, and fetch logs for HPC Jobs. The **Run** button appears only where an administrator has enabled the `ShortLived.EnableHPCJobs` feature flag. Domino gates UI and API submission behind it while the submission experience is refined. Submitting from a Workspace terminal needs no flag. ## Supported HPC Job types | Job type | Example | Notes | | :---------------------------- | :---------------------------------------------- | :-------------------------------------------------------------------------------------------------------- | | Serial shell scripts | `sbatch job.sh` | Resource requests through command-line options or the `#SBATCH` header. | | SMP (single-node, multi-core) | `#SBATCH --cpus-per-task=4`, `#SBATCH --mem=1G` | `--cpus-per-task` sets cores per task, `--mem` sets memory per node. | | Array jobs | `#SBATCH --array=0-9` | Use `$SLURM_ARRAY_JOB_ID` and `$SLURM_ARRAY_TASK_ID` in the script to give each task its own output path. | | Dependent jobs | `sbatch --dependency=afterok: job.sh` | Run a job only after another completes, with `afterok`, `afterany`, or `afternotok`. | ## Monitor and manage HPC Jobs Track jobs from the Workspace terminal with `squeue`, `scontrol show job `, and `sacct`, and cancel them with `scancel `. The Control Plane captures the runtime statistics of every HPC Job and stores them permanently, so the record outlives the job, the worker nodes that ran it, and the Workspace that submitted it. Inside your job scripts, express resource requests in Slurm-native terms: CPUs, GPUs, memory, and wall time. Slurm schedules each job onto workers of the Worker Hardware Tier you selected for the cluster, and Domino autoscales those workers up and down based on the queue. ## Limitations * A Slurm cluster is attached at Workspace launch, and the Workspace is the only place the Slurm CLI runs. A standard Domino Job cannot carry an attached Slurm cluster. * Data moves between the Workspace and the workers only through [Datasets](/6.3/platform-capabilities/core-concepts/data/datasets), a shared network file system. Workers cannot reach [Data Sources](/6.3/platform-capabilities/core-concepts/data/data-source-connectors) or [NetApp Volumes](/6.3/platform-capabilities/core-concepts/data/netapp-volumes). * Interactive and scheduled Slurm jobs are not supported. Jobs from different users can run on the same worker node and are not sandboxed from one another. All Domino users share a single Unix group on the workers, so files written with group-readable or group-writable permissions are visible to other users. Set a restrictive `umask` on sensitive output files. ## Troubleshooting Two conditions must both hold: at least one Environment labeled with the **Slurm** supported-cluster type is available to you, and the Data Plane selected on the **Data Plane & Hardware** tab has Slurm installed. Check your Environment selection and Data Plane, or ask your administrator. Domino overwrites `--output` and `--error` on every HPC Job so that it can collect job output into the system of record, so both resolve to predefined locations and setting them in your script has no effect. To keep your own copy of a job's output, redirect it inside the script to the shared Dataset directory, as the sample job script does. Worker nodes cannot write to your Workspace file system, so any path outside the Dataset fails. Domino autoscales workers on demand, so a cold cluster needs time to provision nodes. If jobs never dispatch, check that your resource request fits within the Worker Hardware Tier and the configured maximum number of workers, or ask your administrator about scheduling limits. ## Related * [Administer Slurm for HPC workloads](/6.3/admin/infrastructure-and-compute/administer-slurm): enable Slurm, create the Environments, and size the worker Hardware Tiers. * [On-demand distributed computing](/6.3/platform-capabilities/features/compute-clusters-hpc): the other cluster types Domino provisions. * [Distributed GPUs with Open MPI](/6.3/platform-capabilities/features/compute-clusters-hpc/mpi): run MPI programs on a Domino-managed MPI cluster. * [Datasets](/6.3/platform-capabilities/core-concepts/data/datasets): the shared storage Slurm jobs read and write. ## Related topics - [Administer Slurm for HPC workloads](/6.3/admin/infrastructure-and-compute/administer-slurm.md) - [Glossary](/6.3/reference/glossary.md) - [Check whether Slurm submission is supported](/6.3/api-reference/hpc-jobs/check-whether-slurm-submission-is-supported.md) - [Start an HPC Job](/6.3/api-reference/hpc-jobs/start-an-hpc-job.md)