> ## Documentation Index
> Fetch the complete documentation index at: https://docs.domino.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Administer Slurm for HPC workloads
> Architecture, setup, and operation of Slurm on a Domino Data Plane, so practitioners can run batch HPC Jobs from a Workspace.
Slurm support adds a persistent Slurm cluster to a Data Plane, with autoscaled worker capacity attached to it, so practitioners can submit batch [HPC Jobs](/cloud/platform-capabilities/features/compute-clusters-hpc/slurm) from a Workspace using the standard Slurm CLI. Domino enables Slurm when a Data Plane is deployed. After that, your work is creating the two Compute Environments Slurm needs, sizing the worker Hardware Tiers, and making sure a shared Dataset is available on the Data Plane.
## How it works
Domino runs High-Performance Computing (HPC) workloads on [Slurm](https://slurm.schedmd.com/), deployed on Kubernetes with [Slinky](https://slinky.schedmd.com/), SchedMD's Slurm operator. Slurm is supported through the Distributed Compute Operator (DCO), like the other compute cluster types, but its deployment model differs in one important way: rather than one ephemeral cluster per execution, each Data Plane runs a single persistent Slurm cluster shared across users, with ephemeral, autoscaled compute attached to it. Slurm is a stateful scheduler, where priority, fair-share, backfill, and limits all depend on recent global state, so its central scheduling and accounting services persist on the Data Plane while worker capacity scales up and down with demand.
* **Slinky** manages all Slurm cluster resources on Kubernetes, including Slurm's central services: the controller, which schedules jobs, and the accounting service, which records the job runtime data used for quotas, access control, and Domino's job monitoring.
* **Slurm is optional**, and adds services to the Data Plane that raise its baseline resource usage.
* **Slurm is part of the Data Plane.** The Control Plane requires no additional components, so a configuration with Slurm absent from the Control Plane and the local Data Plane but installed on a remote Data Plane is valid. Domino enables the Slurm controls in the UI from the component information each Data Plane reports, and the Control Plane collects Slurm job statuses and metrics from the Data Plane no matter how a job was submitted.
* **Runtime statistics are stored permanently.** The Control Plane captures the runtime statistics of every HPC Job and keeps them indefinitely, so the record outlives the job, the worker nodes that ran it, and the Workspace that submitted it.
Workspaces with an attached Slurm cluster act as Slurm login nodes. Domino injects the configuration and credentials the Slurm CLI needs.
### Resources created at runtime
* **Nodesets**: groups of worker nodes with identical characteristics, managed by the DCO. Each Workspace gets its own dedicated nodeset.
* **Partitions**: logical scheduling queues, currently one per nodeset, managed by the Slurm controller through the nodeset custom resource.
* **Workers**: compute nodes, scaled up and down automatically based on queued work. Each worker belongs to exactly one nodeset and one partition.
### Execution flow
When a user launches a Workspace with an attached Slurm cluster:
The dispatcher creates a `slurmcluster` custom resource from the launch parameters and the Project configuration, and sends the manifest to the Data Plane.
The agent applies the custom resource on the Data Plane cluster.
The DCO creates a nodeset resource, from which the Slurm controller creates up to the user's maximum number of worker pods, initially scaled to zero. It creates a token resource, from which the Slurm controller creates a JWT secret for the execution. It then adds the launching user to the `slurm-users` ConfigMap.
## Prerequisites
* Slurm enabled on at least one Data Plane in your deployment. Slurm is enabled when the Data Plane is deployed and cannot be added afterward, so contact Domino Support to enable it.
* A [Dataset](/cloud/platform-capabilities/core-concepts/data/datasets) configured on the Slurm-enabled Data Plane. See [Configure a shared Dataset](#configure-a-shared-dataset).
## Create the Compute Environments
Slurm requires two [Compute Environments](/cloud/platform-capabilities/core-concepts/compute-environments): one with the Slurm CLI installed, which users select for the Workspace itself, and one that runs on the Slurm worker nodes. Create both by [creating an Environment](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/3-create-an-environment) with the attributes below. Get the current image tags from the [Compute Environment catalog](/cloud/platform-capabilities/core-concepts/compute-environments/manage-compute-environments/8-compute-environment-catalog).
### Workspace Compute Environment
* **Environment Base**: **Start from a custom base image**, then enter the Slurm Workspace image.
```
quay.io/domino/domino-slurm-environment:
```
* **Supported Clusters**: **None**. This Environment runs the Workspace, not the workers.
* **Pluggable Workspace Tools**: add the workspace tool definitions.
```yaml theme={null}
jupyter:
title: "Jupyter (Python, R, Julia)"
iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
start: [ "/opt/domino/workspaces/jupyter/start" ]
supportedFileExtensions: [ ".ipynb" ]
httpProxy:
port: 8888
rewrite: false
internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
requireSubdomain: false
jupyterlab:
title: "JupyterLab"
iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
start: [ "/opt/domino/workspaces/jupyterlab/start" ]
httpProxy:
internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
port: 8888
rewrite: false
requireSubdomain: false
vscode:
title: "vscode"
iconUrl: "/assets/images/workspace-logos/vscode.svg"
start: [ "/opt/domino/workspaces/vscode/start" ]
httpProxy:
port: 8888
requireSubdomain: false
rstudio:
title: "RStudio"
iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
start: [ "/opt/domino/workspaces/rstudio/start" ]
httpProxy:
port: 8888
requireSubdomain: false
```
### Cluster Compute Environment
* **Environment Base**: **Start from a custom base image**, then enter the Slurm cluster image.
```
quay.io/domino/slurm-cluster-environment:
```
* **Supported Clusters**: **Slurm**.
The **Slurm** option in the Workspace launcher is enabled only when at least one Environment carrying the Slurm supported-cluster label is available to the user. Make both Environments visible to your intended users or their Organizations.
## Create worker Hardware Tiers
Users select the worker tier that best fits their jobs when they attach a Slurm cluster to a Workspace. A typical deployment defines a small set of [Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers) matched to common workload profiles:
| Hardware Tier | Workload profile | Backing Kubernetes nodes |
| :--------------- | :------------------------------------------------ | :--------------------------------------------- |
| Slurm Compute | CPU-bound: simulations and parameter sweeps | Standard nodes with a high CPU-to-memory ratio |
| Slurm Big Memory | Memory-bound: large in-memory models and datasets | High-memory nodes |
| Slurm GPU | GPU-bound: training, GPU-accelerated simulation | GPU-enabled nodes |
For predictable scheduling, size each Hardware Tier to match its underlying Kubernetes node. That avoids one Kubernetes node hosting several Slurm workers, and a Slurm worker sharing a node with unrelated workloads.
When sizing, leave headroom for the small per-node overhead that infrastructure `DaemonSets` add. For a Kubernetes node with 8 CPUs and 32 GiB of memory, for example, a worker Hardware Tier of 6 CPUs and 28 GiB works well.
## Configure a shared Dataset
Slurm has no internal mechanism for copying files between the Workspace and the worker nodes, so data is exchanged through a shared network file system. In Domino that is a [Dataset](/cloud/platform-capabilities/core-concepts/data/datasets), which makes configuring one on the Slurm-enabled Data Plane essential rather than optional.
* **Local Data Plane**: configured by default at `/domino/datasets/local/`.
* **Remote Data Plane**: enable Data Plane storage and Workspaces on the Data Plane. See [Manage Data Planes](/cloud/admin/platform-configuration/manage-data-planes).
## Set scheduling policies and limits
Configure Slurm's supported scheduling policies and limits, including priority, fair-share, and per-user and per-Project limits on queued or running jobs, directly in Slurm with the standard Slurm administration tools. The entry point for CLI-based Slurm configuration is a Workspace on the Slurm-enabled Data Plane. The [Slurm documentation](https://slurm.schedmd.com/) covers the available policy and limit options.
## Enable HPC Job submission from the Jobs page
The **HPC** tab on a Project's Jobs page carries the history of the Project's HPC Jobs, and lists them no matter how they were submitted. Submission from that tab is gated behind the `ShortLived.EnableHPCJobs` feature flag. Set it to `true` and a **Run** button appears on the tab, letting users submit HPC Jobs from the Domino UI and the API rather than from a Workspace terminal. Domino ships the gate because the submission experience is still being refined. Submission from a Workspace terminal is unaffected and needs no flag.
## Map Domino users to Slurm users
Each Slurm user name derives from the name of the Domino user who launched the Workspace, so Slurm accounting, fair-share, and per-user limits apply per person rather than to one shared account.
A Domino user name has to be a valid Linux user name for this mapping to work: a lowercase letter or underscore followed by up to 31 lowercase letters, digits, underscores, or hyphens, matching `^[a-z_][a-z0-9_-]{0,31}`. It must also not be a reserved system name such as `root`, `daemon`, or `nobody`.
Launching a Workspace with an attached Slurm cluster fails when the Domino user name breaks either rule. Check user names against both before you make Slurm available to a group of users.
A Domino user holding the SysAdmin role maps instead to Slurm's built-in `slurm` user, which is a Slurm system administrator. The [SysAdmin role](/cloud/admin/identity-and-access/manage-users/roles) is not available on Domino Cloud, so this mapping applies only to self-managed deployments.
## Limitations
* A Slurm cluster is attached at Workspace launch, and the Workspace is the only place the Slurm CLI runs. A standard Domino Job cannot carry an attached Slurm cluster.
* Nodesets and partitions are created per Workspace, so each Workspace gets its own dedicated nodeset.
* Data is exchanged between the Workspace and the workers only through Datasets, over NFS. Workers cannot reach [Data Sources](/cloud/platform-capabilities/core-concepts/data/data-source-connectors) or [NetApp Volumes](/cloud/platform-capabilities/core-concepts/data/netapp-volumes).
* Integration with external, customer-managed Slurm clusters is not supported.
* Slurm cannot be added to a Data Plane that is already deployed.
Jobs on worker nodes are not sandboxed from one another. Jobs from different users can run on the same worker node, and all Domino users share a single Unix group there, so files written with group-readable or group-writable permissions are visible to other users. Tell users to set a restrictive `umask` on sensitive output files.
## Known issues
### Slurm REST API memory growth
Broad `sacct` queries, such as `sacct -a`, run after high job volume cause sustained memory growth in the Slurm REST API. Memory is not promptly released, which can lead to out-of-memory restarts of the service.
To mitigate it, tell users to filter `sacct` queries by job ID, user, or time range rather than querying all jobs. If the service restarts repeatedly, contact Domino Support.
## Troubleshooting
Two conditions must both hold: at least one Environment labeled with the **Slurm** supported-cluster type is available to the user, and the Data Plane they selected has Slurm installed.
Standard Jobs do not support attached Slurm clusters. Users submit HPC Jobs from a Workspace terminal, or from the **HPC** tab of the Project's Jobs page once you turn on the flag described in [Enable HPC Job submission from the Jobs page](#enable-hpc-job-submission-from-the-jobs-page).
Confirm that the requested resources fit within the worker Hardware Tier and the maximum number of workers set on the cluster. If workers are never created at all, contact Domino Support.
## Related
* [Run HPC workloads with Slurm](/cloud/platform-capabilities/features/compute-clusters-hpc/slurm): the practitioner-facing guide to submitting HPC Jobs.
* [Manage Hardware Tiers](/cloud/admin/infrastructure-and-compute/manage-compute-resources/hardware-tiers): create and size the worker tiers.
* [Manage Data Planes](/cloud/admin/platform-configuration/manage-data-planes): register a Data Plane and enable it for storage and Workspaces.
* [Datasets](/cloud/platform-capabilities/core-concepts/data/datasets): the shared storage Slurm jobs read and write.
## Related topics
- [Run HPC workloads with Slurm](/cloud/platform-capabilities/features/compute-clusters-hpc/slurm/index.md)
- [Domino infrastructure](/cloud/admin/infrastructure-and-compute/index.md)
- [Glossary](/cloud/reference/glossary.md)
- [On-demand distributed computing](/cloud/platform-capabilities/features/compute-clusters-hpc/index.md)