Kubernetes infrastructure

Domino installs into a Kubernetes cluster that you provide, and you manage the deployment with standard Kubernetes tooling such as kubectl. Domino uses two types of Kubernetes node:

Node type	Label	Hosts	Scaling
Platform	`dominodatalab.com/platform-node: true`	The always-on components of the Domino application, including the frontends, API server, authentication service, and supporting metadata services	A fixed collection of persistent pods
Compute	`dominodatalab.com/node-pool:<node name>`	User Jobs and published Domino endpoints and Apps	Changes with user demand, so an elastic cloud cluster can scale the pool automatically

Run platform and compute nodes in separate node pools wherever your infrastructure allows it. Domino sets resource limits and requests either way, so no single workload can overwhelm a node. See Domino architecture for the services that run on each node type.

Kubernetes compatibility

Domino validates its software on Amazon Elastic Kubernetes Service (EKS), Azure Kubernetes Service (AKS), Google Kubernetes Engine (GKE), and Red Hat OpenShift. Domino also validates non-vCluster Nexus data planes on Amazon EKS. For security and stability, keep your Kubernetes infrastructure current and run a version of Domino that Domino has validated against it. Outdated or unvalidated combinations of Domino and Kubernetes can affect the supportability of your deployment.

Compatibility matrix

The following matrix shows the supported combinations of Domino and Kubernetes platform versions. To ask about an unlisted Kubernetes distribution, contact your Domino professional services representative. The Kubernetes platform provider dictates end of life (EOL) for Kubernetes versions. For what EOL means on each platform, see the links in the Kubernetes platform column.

	Control plane Domino version
Kubernetes platform	6.3.0
Amazon Elastic Kubernetes Service (EKS)	1.34, 1.35
Azure Kubernetes Service (AKS)	1.34, 1.35
Google Kubernetes Engine (GKE)	1.34, 1.35
Red Hat OpenShift	1.34 (4.21)
Nexus data plane (non-vCluster \| AWS (EKS))	1.34, 1.35

Upgrade guidance

Kubernetes versions reach EOL every 12 to 14 months, after which the Kubernetes provider may stop supporting them. Align your installation and upgrade planning with the following guidance, and contact Domino customer support with questions about compatibility and upgrade paths.

Domino ships major or minor releases approximately every 3 months. Upgrade every 3 to 6 months to stay current. Skipping multiple versions of Domino produces larger gaps in Kubernetes version compatibility and more complex upgrades.
Each Kubernetes provider has a different release schedule for specific versions of Kubernetes. Domino aims to support new Kubernetes releases within approximately 3 months of their generally available release date.
Upgrade Domino onto a supported Kubernetes version. When you upgrade Kubernetes, upgrade to a supported Domino version immediately afterwards.
Keep the Kubernetes control plane and data plane at the same version to minimize version skew.

Domino Nexus hybrid deployments

Nexus Control Planes have the same Kubernetes cluster requirements as any other Domino deployment. Nexus Data Planes differ in four ways:

They need no shared storage (no ReadWriteMany storage class).
They need only one namespace, for compute.
Their ingress configuration differs. See Enable a Data Plane for Workspaces.
Their Kubernetes version compatibility can differ from the control plane. See the compatibility matrix.

Docker compatibility

Domino doesn’t publish a validated container runtime compatibility matrix and recommends the default container runtime version your vendor provides. If your Kubernetes nodes use Docker, use 20.10.24 or higher.

Cluster requirements

Deploy Domino into a Kubernetes cluster that meets the requirements in this section.

Cluster permissions

Domino needs permission to install and configure pods in the cluster through Helm. The Domino installer is a containerized Python utility that operates Helm through a kubeconfig that provides service account access to the cluster.

Namespaces

Domino creates one dedicated namespace for platform nodes, one for compute nodes, and one for installer metadata and secrets.

Node pool requirements

Domino requires at least two node pools, one for the Domino platform and one for compute workloads. Add optional pools to provide specialized execution hardware for some compute workloads. Domino recommends starting with four platform nodes and two compute nodes. The table below gives each pool’s floor and ceiling rather than that starting size, so the compute pool’s minimum is lower than the two nodes Domino recommends you begin with.

Pool	Min nodes	Max nodes	Spec	Boot disk	Labels
Platform	4	4	8 CPU / 32 GB	128 GB	`dominodatalab.com/node-pool: platform`
Compute	1 (recommended)	As many as demand requires	8 CPU / 32 GB (recommended minimum)	400 GB	`dominodatalab.com/node-pool: default`, `domino/build-node: true`
GPU (optional)	0 (recommended)	As many as demand requires	8 CPU / 16 GB plus one or more NVIDIA GPU devices (recommended minimum)	400 GB	`dominodatalab.com/node-pool: default-gpu`, `nvidia.com/gpu: true`

Enable autoscaling on the compute pool. Node requirements increased in Domino 5.0, so if you upgrade to 5.0 or higher and the platform cluster does not support autoscaling, raise the node count to cover the requirements above. Tag every pool with kubernetes.io/cluster/{{ cluster_name }}: owned. For autoscaler autodiscovery, also tag every pool with k8s.io/cluster-autoscaler/enabled: true and k8s.io/cluster-autoscaler/{{ cluster_name }}: owned. The compute and GPU pools carry these additional tags:

Pool	Additional tags
Compute	`k8s.io/cluster-autoscaler/node-template/label/dominodatalab.com/node-pool: default`, `k8s.io/cluster-autoscaler/node-template/label/domino/build-node: true`
GPU	`k8s.io/cluster-autoscaler/node-template/label/dominodatalab.com/node-pool: default-gpu`

Configure GPU nodes with the appropriate NVIDIA driver and nvidia-docker2, and set the default Docker runtime to nvidia. The EKS GPU-optimized Amazon Machine Image (AMI) is one example.

Storage requirements

Storage classes

Domino requires at least two storage classes.

Storage class	Default name	Access mode	Backs
Dynamic block storage	`dominodisk`	`ReadWriteOnce`	Ephemeral volumes attached to user executions, and the high-performance databases that hold Domino application object data
Long-term shared storage	`dominoshared`	`ReadWriteMany`	Project data uploaded or created by users, Domino Datasets, Docker images, and Domino backups

Dynamic block storage must be backed by a storage class that:

Supports dynamic provisioning
Can be mounted on any node in the cluster
Can provision volumes of at least 100 GB
Supports ReadWriteOnce semantics
Is backed by true, fully POSIX-compliant block storage, not Network File System (NFS)

Domino recommends solid-state backing for fast I/O.

If this storage does not meet these requirements, or if you override critical services that rely on block storage (MongoDB, Postgres, Git) to use a different storage class, you may see performance degradations, catastrophic failures, and unexpected data loss.

Provision approximately 3 TB of block storage from non-cloud storage providers such as EBS, Longhorn, or Hitachi. (This requirement does not apply to cloud providers such as EKS, AKS, and GKE.)

The following examples show compatible block storage classes on two platforms.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: domino-compute-storage
provisioner: kubernetes.io/aws-ebs
parameters:
  type: gp2
  fsType: ext4

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: dominodisk
parameters:
  replication-type: none
  type: pd-standard
provisioner: kubernetes.io/gce-pd
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Long-term shared storage must be backed by a storage class that:

Dynamically provisions Kubernetes PersistentVolumes
Can be accessed in ReadWriteMany mode from all nodes in the cluster
Uses a volumeBindingMode of Immediate

In AWS, for example, one class backed by Elastic File System (EFS) covers Domino Datasets, and one class backed by S3 covers project data, backups, and Docker images.

Storage by resource

Domino uses cloud-provider native object storage where it is available, and the dominoshared storage class everywhere else.

Resource	AWS	Other infrastructure
Blob storage	S3. See Blob storage.	The `dominoshared` storage class
Logs	S3. See Blob storage.	The `dominoshared` storage class
Costs	S3. See Blob storage.	See the note below
Datasets	EFS. See Datasets storage.	The `dominoshared` storage class
Backups	S3	Azure Blob Storage, Google Cloud Storage, or the `dominoshared` storage class on-premises

On-premises storage

On-premises deployments use a wide variety of block and file-based storage. Back dominodisk with block storage rather than NFS, ideally meeting the dynamic block storage requirements above. In some cases host volumes can back services such as Git, Postgres, and MongoDB, and Postgres and MongoDB replicate their own state. Host volumes can also back Runs, though network-attached block storage is preferred because it keeps cached files portable between nodes. If you use host volumes for Runs, disable file caching and expect slow start-up executions for large projects.

Cluster networking

Domino relies on Kubernetes network policies to manage secure communication between pods, so your networking solution must support the Kubernetes NetworkPolicy resource. Calico is one such solution.

Ingress and SSL

Configure Domino to serve from a specific fully qualified domain name (FQDN), and resolve DNS for that name to the address of an SSL-terminating load balancer with a valid certificate.

A Domino install can’t be hosted on a subdomain of another Domino install. For example, if you have Domino deployed at data-science.example.com, you can’t deploy another instance of Domino at acme.data-science.example.com.

The load balancer must target incoming connections on ports 80 and 443 to port 80 on all nodes in the platform pool, and it must support WebSocket connections. Health checks must use HTTP on port 80 and check for 200 responses from a path of /healthz. So that Domino detects the protocol of incoming requests correctly, the SSL-terminating load balancer must set the X-Forwarded-Proto header. Domino does not support the alternative Forwarded header.

Domino continues to support Environments with subdomains, but recommends against them for improved security. If you use subdomains for your Domino deployment and need best-practice information, contact your account manager.

NTP

The nodes in your Kubernetes cluster must have a valid Network Time Protocol (NTP) configuration. Accurate clocks are a prerequisite for single sign-on (SSO) protocols, Transport Layer Security (TLS) connections to external services, intra-cluster TLS when you use Istio, and other time-sensitive protocols. They also prevent TLS validation failures and general interoperability issues.

NVIDIA DGX systems

NVIDIA DGX systems run Domino workloads once you add them to your Kubernetes cluster as compute (worker) nodes.

Swim lane flow diagram with three lanes. Domino Administrator: a scientist or researcher requests a GPU-enabled Hardware Tier, and once a DGX is a worker node in the cluster the administrator configures a GPU Hardware Tier from the DGX-enabled node pool, configures a Compute Environment with the required tooling, and deploys a Workspace. Kubernetes administrator: receives the request, checks whether a DGX is available in the IT stack and whether the node has Kubernetes API access, then configures host access and sets node labels and a GPU taint. Procurement and Central IT Management: purchases the DGX, installs it in the on-premises IT environment, and configures networking from the DGX host to the Kubernetes cluster.

Three groups share the work. Procurement and central IT purchase the DGX, install it, and connect it to the cluster network. A Kubernetes administrator adds the node, sets its labels, and applies a GPU taint. A Domino administrator then builds a GPU Hardware Tier on that node pool, so practitioners can request it.

Install a DGX system

Purchase and provision the DGX system

Purchase NVIDIA DGX systems through NVIDIA’s Partner Network. Install the system in a hosting environment with network access to the node and storage infrastructure that Domino requires.

Add the DGX system to a Kubernetes cluster

Existing cluster
New cluster

Add the DGX to your Kubernetes API server as a worker node, with a node label that matches your naming conventions. The default node label for GPU-based worker nodes is default-gpu.
Add taints to the DGX node so that Domino selects it for GPU-based workloads.

Configure a GPU-enabled Hardware Tier

Once the DGX is added to your API server and labeled, configure Hardware Tiers in the Domino admin application, and set user execution quotas to control who consumes the hardware.

Configure CUDA and NVIDIA drivers

Install the NVIDIA driver on the host

Your server administrator configures the NVIDIA driver at the host level. Use NVIDIA’s driver downloads to identify the correct driver for your host, and see the DGX systems documentation for more detail.

Identify the CUDA version your framework requires

Each development framework documents the CUDA software version it needs. For example, TensorFlow 2.1 and later requires CUDA 10.1 plus additional packages such as cuDNN.

Match the CUDA and driver versions

Consult the CUDA and NVIDIA driver compatibility table. In the TensorFlow 2.1 example, CUDA 10.1 means the host must run CUDA 10.1 or later and NVIDIA driver 410.48 or later.

Set the CUDA version in the Compute Environment

Configure the Compute Environment to use the exact CUDA version the application needs. CUDA drivers are backwards compatible, so the host’s CUDA version can be greater than or equal to the version the Compute Environment specifies.Installing an exact CUDA version, patch version included, often returns unexpected results. The fastest route to a working configuration is usually to install the latest minor release of the major CUDA version you need, then set a Docker environment variable in your Compute Environment that constrains the compatible set of CUDA versions, GPU generations, and NVIDIA drivers.

For guidance on your specific needs, consult your Domino customer success engineer, who can share sample configurations that simplify the process.

DGX best practices

Keep DGX GPUs off build nodes. Use a CPU resource as the build node for Compute Environments instead.
Split GPUs across several Hardware Tiers. Offer tiers of 1, 2, 4, and 8 GPUs, for example. Training Jobs use single or parallel GPUs, and consuming a whole DGX for one workload is often not feasible.
Restrict access per organization. After splitting the tiers, grant access globally or limit it to specific organizations. Give the organizations that need GPUs access and restrict the rest, so critical work keeps its capacity and no one consumes a GPU tier without authorization.

Next steps

Set up Domino on EKS, AKS, GKE, or an on-premises cluster.
Deploy workloads across multiple Kubernetes clusters with Nexus hybrid architecture.
Work through the Domino installation process.

Architecture & deployment

Domino installation

Identity & access

Infrastructure & compute

Platform configuration

Data administration

Operations, monitoring & cost

Security & compliance

Troubleshooting

Kubernetes infrastructure

Kubernetes compatibility

Compatibility matrix

Upgrade guidance

Domino Nexus hybrid deployments

Docker compatibility

Cluster requirements

Cluster permissions

Namespaces

Node pool requirements

Storage requirements

Storage classes

Storage by resource

On-premises storage

Cluster networking

Ingress and SSL

NTP

NVIDIA DGX systems

Install a DGX system

Configure CUDA and NVIDIA drivers

DGX best practices

Next steps

​Kubernetes compatibility

​Compatibility matrix

​Upgrade guidance

​Domino Nexus hybrid deployments

​Docker compatibility

​Cluster requirements

​Cluster permissions

​Namespaces

​Node pool requirements

​Storage requirements

​Storage classes

​Storage by resource

​On-premises storage

​Cluster networking

​Ingress and SSL

​NTP

​NVIDIA DGX systems

​Install a DGX system

​Configure CUDA and NVIDIA drivers

​DGX best practices

​Next steps

Kubernetes compatibility

Compatibility matrix

Upgrade guidance

Domino Nexus hybrid deployments

Docker compatibility

Cluster requirements

Cluster permissions

Namespaces

Node pool requirements

Storage requirements

Storage classes

Storage by resource

On-premises storage

Cluster networking

Ingress and SSL

NTP

NVIDIA DGX systems

Install a DGX system

Configure CUDA and NVIDIA drivers

DGX best practices

Next steps