Administer Slurm for HPC workloads

Slurm support adds a persistent Slurm cluster to a Data Plane, with autoscaled worker capacity attached to it, so practitioners can submit batch HPC Jobs from a Workspace using the standard Slurm CLI. Domino enables Slurm when a Data Plane is deployed. After that, your work is creating the two Compute Environments Slurm needs, sizing the worker Hardware Tiers, and making sure a shared Dataset is available on the Data Plane.

​

How it works

Domino runs High-Performance Computing (HPC) workloads on Slurm, deployed on Kubernetes with Slinky, SchedMD’s Slurm operator. Slurm is supported through the Distributed Compute Operator (DCO), like the other compute cluster types, but its deployment model differs in one important way: rather than one ephemeral cluster per execution, each Data Plane runs a single persistent Slurm cluster shared across users, with ephemeral, autoscaled compute attached to it. Slurm is a stateful scheduler, where priority, fair-share, backfill, and limits all depend on recent global state, so its central scheduling and accounting services persist on the Data Plane while worker capacity scales up and down with demand.

• Slinky manages all Slurm cluster resources on Kubernetes, including Slurm’s central services: the controller, which schedules jobs, and the accounting service, which records the job runtime data used for quotas, access control, and Domino’s job monitoring.
• Slurm is optional, and adds services to the Data Plane that raise its baseline resource usage.
• Slurm is part of the Data Plane. The Control Plane requires no additional components, so a configuration with Slurm absent from the Control Plane and the local Data Plane but installed on a remote Data Plane is valid. Domino enables the Slurm controls in the UI from the component information each Data Plane reports, and the Control Plane collects Slurm job statuses and metrics from the Data Plane no matter how a job was submitted.
• Runtime statistics are stored permanently. The Control Plane captures the runtime statistics of every HPC Job and keeps them indefinitely, so the record outlives the job, the worker nodes that ran it, and the Workspace that submitted it.

Workspaces with an attached Slurm cluster act as Slurm login nodes. Domino injects the configuration and credentials the Slurm CLI needs.

​

Resources created at runtime

• Nodesets: groups of worker nodes with identical characteristics, managed by the DCO. Each Workspace gets its own dedicated nodeset.
• Partitions: logical scheduling queues, currently one per nodeset, managed by the Slurm controller through the nodeset custom resource.
• Workers: compute nodes, scaled up and down automatically based on queued work. Each worker belongs to exactly one nodeset and one partition.

​

Execution flow

When a user launches a Workspace with an attached Slurm cluster:

​

Prerequisites

• Slurm enabled on at least one Data Plane in your deployment. Slurm is enabled when the Data Plane is deployed and cannot be added afterward, so contact Domino Support to enable it.
• A Dataset configured on the Slurm-enabled Data Plane. See Configure a shared Dataset.

​

Create the Compute Environments

Slurm requires two Compute Environments: one with the Slurm CLI installed, which users select for the Workspace itself, and one that runs on the Slurm worker nodes. Create both by creating an Environment with the attributes below. Get the current image tags from the Compute Environment catalog.

​

Workspace Compute Environment

• Environment Base: Start from a custom base image, then enter the Slurm Workspace image.
  quay.io/domino/domino-slurm-environment:<tag>
  
• Supported Clusters: None. This Environment runs the Workspace, not the workers.
• Pluggable Workspace Tools: add the workspace tool definitions.
  jupyter:
    title: "Jupyter (Python, R, Julia)"
    iconUrl: "/assets/images/workspace-logos/Jupyter.svg"
    start: [ "/opt/domino/workspaces/jupyter/start" ]
    supportedFileExtensions: [ ".ipynb" ]
    httpProxy:
      port: 8888
      rewrite: false
      internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
      requireSubdomain: false
  jupyterlab:
    title: "JupyterLab"
    iconUrl: "/assets/images/workspace-logos/jupyterlab.svg"
    start: [ "/opt/domino/workspaces/jupyterlab/start" ]
    httpProxy:
      internalPath: "/{{ownerUsername}}/{{projectName}}/{{sessionPathComponent}}/{{runId}}/{{#if pathToOpen}}tree/{{pathToOpen}}{{/if}}"
      port: 8888
      rewrite: false
      requireSubdomain: false
  vscode:
    title: "vscode"
    iconUrl: "/assets/images/workspace-logos/vscode.svg"
    start: [ "/opt/domino/workspaces/vscode/start" ]
    httpProxy:
      port: 8888
      requireSubdomain: false
  rstudio:
    title: "RStudio"
    iconUrl: "/assets/images/workspace-logos/Rstudio.svg"
    start: [ "/opt/domino/workspaces/rstudio/start" ]
    httpProxy:
      port: 8888
      requireSubdomain: false
  

​

Cluster Compute Environment

• Environment Base: Start from a custom base image, then enter the Slurm cluster image.
  quay.io/domino/slurm-cluster-environment:<tag>
  
• Supported Clusters: Slurm.

​

Create worker Hardware Tiers

Users select the worker tier that best fits their jobs when they attach a Slurm cluster to a Workspace. A typical deployment defines a small set of Hardware Tiers matched to common workload profiles: For predictable scheduling, size each Hardware Tier to match its underlying Kubernetes node. That avoids one Kubernetes node hosting several Slurm workers, and a Slurm worker sharing a node with unrelated workloads. When sizing, leave headroom for the small per-node overhead that infrastructure DaemonSets add. For a Kubernetes node with 8 CPUs and 32 GiB of memory, for example, a worker Hardware Tier of 6 CPUs and 28 GiB works well.

​

Configure a shared Dataset

Slurm has no internal mechanism for copying files between the Workspace and the worker nodes, so data is exchanged through a shared network file system. In Domino that is a Dataset, which makes configuring one on the Slurm-enabled Data Plane essential rather than optional.

• Local Data Plane: configured by default at /domino/datasets/local/<project-name>.
• Remote Data Plane: enable Data Plane storage and Workspaces on the Data Plane. See Manage Data Planes.

​

Set scheduling policies and limits

Configure Slurm’s supported scheduling policies and limits, including priority, fair-share, and per-user and per-Project limits on queued or running jobs, directly in Slurm with the standard Slurm administration tools. The entry point for CLI-based Slurm configuration is a Workspace on the Slurm-enabled Data Plane. The Slurm documentation covers the available policy and limit options.

​

Enable HPC Job submission from the Jobs page

The HPC tab on a Project’s Jobs page carries the history of the Project’s HPC Jobs, and lists them no matter how they were submitted. Submission from that tab is gated behind the ShortLived.EnableHPCJobs feature flag. Set it to true and a Run button appears on the tab, letting users submit HPC Jobs from the Domino UI and the API rather than from a Workspace terminal. Domino ships the gate because the submission experience is still being refined. Submission from a Workspace terminal is unaffected and needs no flag.

​

Map Domino users to Slurm users

Each Slurm user name derives from the name of the Domino user who launched the Workspace, so Slurm accounting, fair-share, and per-user limits apply per person rather than to one shared account. A Domino user name has to be a valid Linux user name for this mapping to work: a lowercase letter or underscore followed by up to 31 lowercase letters, digits, underscores, or hyphens, matching ^[a-z_][a-z0-9_-]{0,31}. It must also not be a reserved system name such as root, daemon, or nobody. A Domino user holding the SysAdmin role maps instead to Slurm’s built-in slurm user, which is a Slurm system administrator. The SysAdmin role is not available on Domino Cloud, so this mapping applies only to self-managed deployments.

​

Limitations

• A Slurm cluster is attached at Workspace launch, and the Workspace is the only place the Slurm CLI runs. A standard Domino Job cannot carry an attached Slurm cluster.
• Nodesets and partitions are created per Workspace, so each Workspace gets its own dedicated nodeset.
• Data is exchanged between the Workspace and the workers only through Datasets, over NFS. Workers cannot reach Data Sources or NetApp Volumes.
• Integration with external, customer-managed Slurm clusters is not supported.
• Slurm cannot be added to a Data Plane that is already deployed.

​

Known issues

​

Slurm REST API memory growth

Broad sacct queries, such as sacct -a, run after high job volume cause sustained memory growth in the Slurm REST API. Memory is not promptly released, which can lead to out-of-memory restarts of the service. To mitigate it, tell users to filter sacct queries by job ID, user, or time range rather than querying all jobs. If the service restarts repeatedly, contact Domino Support.

​

Troubleshooting

​

Related

• Run HPC workloads with Slurm: the practitioner-facing guide to submitting HPC Jobs.
• Manage Hardware Tiers: create and size the worker tiers.
• Manage Data Planes: register a Data Plane and enable it for storage and Workspaces.
• Datasets: the shared storage Slurm jobs read and write.