Install a Data Plane

Installing a Data Plane is the first half of bringing one up. You provision a Kubernetes cluster, install the Domino services onto it, and verify that the Data Plane agent is waiting for its bootstrap. The second half, registering the Data Plane, connects it to the control plane and is where this page ends. Starting with Domino 6.1, Data Planes are installed with the Platform Operator, the same deployment model the control plane uses. That gives both planes one configuration format and one upgrade path.

Set the FLEETCOMMAND_AGENT_TAG and the Data Plane configuration correctly before you apply them. Incorrect values cannot be fixed or updated once the install completes: the deployment is left partially broken and recovers only through manual intervention.

Prerequisites

A functional control plane running Domino 6.1 or later, with Nexus enabled.
A Kubernetes cluster for the Data Plane. See Provision the cluster.

Network connectivity from the Data Plane cluster to four control plane endpoints:

Endpoint	Example	Purpose
Control plane container registry	`registry.domino.example.com`	Pull container images
Control plane hostname	`domino.example.com`	API communication
RabbitMQ service	`rabbit.domino.example.com`	Message queue
Vault service	`vault.domino.example.com`	Secrets management

Configuration records for the RabbitMQ, Vault, and Docker registry hosts, plus any custom ports.
A Domain Name System (DNS) subdomain of your control plane domain. If the control plane is example.com, the Data Plane is something like dp.example.com.
A valid Transport Layer Security (TLS) certificate for the Data Plane hostname.
Registry credentials for quay.io, provided by your Domino account team.

Install these tools on the workstation you run the installation from:

Tool	Purpose
`ddlctl`	The Domino deployer command line. See Install ddlctl.
`kubectl`	Applies and inspects Kubernetes resources on the Data Plane cluster.
Docker	Certain `ddlctl` operations require access to the Docker Engine API. See Docker engine.
`curl`	Downloads the bootstrap script during registration.

Cluster node pools

The Data Plane cluster needs at least two node pools, labeled so Domino can schedule against them:

Node pool	Label	Runs	Scaling
Platform	`dominodatalab.com/node-pool: platform`	Data Plane services	0-1
Compute	`dominodatalab.com/node-pool: default`	User workloads	0-n

Size the nodes for your cloud provider:

Cloud provider	Platform pool	Compute pool
Amazon Elastic Kubernetes Service (EKS)	`m5.2xlarge` or similar	Based on workload needs
Azure Kubernetes Service (AKS)	`Standard_D8s_v5`	`Standard_D16s_v5`
Google Kubernetes Engine (GKE)	`n1-standard-8` or similar	Based on workload needs

On AKS, spread the compute pool across all 3 availability zones by creating one pool per zone. The platform pool needs no zonal distribution, because it holds no persistent storage. The Data Plane cluster’s Kubernetes version can differ from the control plane’s, as long as it sits in the compatible range for your Domino version.

Provision the cluster

Provision an EKS, AKS, or GKE cluster that meets the node pool requirements above, then point your default Kubernetes context at it. For EKS, Domino publishes a Terraform module that provisions the whole stack. One difference from a control plane deployment is worth calling out before you start: a Data Plane does not own the S3 bucket or the Elastic Container Registry (ECR), which stay with the control plane. Deploy Domino on AWS EKS covers the same data-plane-eks preset from the control plane installer’s perspective, including the Terraform values to pull into the configuration file.

Sample AKS node pool configuration

Create one platform pool and one compute pool per availability zone. For the compute pools, name them compute1, compute2, and compute3, and set availabilityZones to ["1"], ["2"], and ["3"] respectively.

{
  "name": "platform",
  "vmSize": "Standard_D8s_v5",
  "osDiskSizeGB": 128,
  "osDiskType": "Ephemeral",
  "kubeletDiskType": "OS",
  "maxPods": 110,
  "type": "VirtualMachineScaleSets",
  "maxCount": 1,
  "minCount": 0,
  "enableAutoScaling": true,
  "scaleDownMode": "Delete",
  "enableNodePublicIP": false,
  "nodeLabels": {
    "dominodatalab.com/node-pool": "platform"
  },
  "mode": "User",
  "osType": "Linux",
  "osSKU": "Ubuntu"
}

{
  "name": "compute1",
  "vmSize": "Standard_D16s_v3",
  "osDiskSizeGB": 128,
  "osDiskType": "Ephemeral",
  "kubeletDiskType": "OS",
  "maxPods": 110,
  "type": "VirtualMachineScaleSets",
  "maxCount": 3,
  "minCount": 0,
  "enableAutoScaling": true,
  "scaleDownMode": "Delete",
  "availabilityZones": ["1"],
  "enableNodePublicIP": false,
  "nodeLabels": {
    "dominodatalab.com/node-pool": "default"
  },
  "mode": "User",
  "osType": "Linux",
  "osSKU": "Ubuntu"
}

Provision an EKS cluster with Terraform

The terraform-aws-eks module uses a three-stage deployment driven by a tf.sh wrapper script. Apply the stages in order, because each depends on the outputs of the one before it:

Stage	What it creates	`.tfvars` file
`infra`	Virtual Private Cloud (VPC), subnets, Key Management Service (KMS) key, bastion, Secure Shell (SSH) key pair	`terraform/infra.tfvars`
`cluster`	EKS control plane, Calico Container Network Interface (CNI), Identity and Access Management (IAM) Roles for Service Accounts (IRSA)	`terraform/cluster.tfvars`
`nodes`	EKS-managed node groups	`terraform/nodes.tfvars`

Beyond the tools listed under Prerequisites, Terraform provisioning needs Terraform 1.3 or later, the AWS CLI v2, Helm 3.x, and hcledit on your PATH. Configure AWS credentials with permission to create VPC resources, EKS clusters, EC2 instances, IAM roles, and security groups, then confirm the right account is active:

aws sts get-caller-identity

The command returns your UserId, Account, and Arn.

Bootstrap the deployment directory

Rather than cloning the full repository, use terraform init -from-module to pull the examples/deploy scaffold at a specific version tag into a local directory. That also runs set-mod-version.sh, which pins all three module sources to the same tag.

Install hcledit, which set-mod-version.sh requires:

brew install minamijoyo/hcledit/hcledit

go install github.com/minamijoyo/hcledit@latest

Set MOD_VERSION to the module version validated for your Domino release, listed on Provision Terraform infrastructure for AWS, then bootstrap the directory:

export MOD_VERSION='<validated module version>'
export DEPLOY_DIR='domino-deploy'

mkdir -p "$DEPLOY_DIR"
terraform -chdir="$DEPLOY_DIR" init -backend=false \
  -from-module="github.com/dominodatalab/terraform-aws-eks.git//examples/deploy?ref=${MOD_VERSION}"

cd $DEPLOY_DIR
unset DEPLOY_DIR
./set-mod-version.sh "$MOD_VERSION"

Run every later ./tf.sh command from this domino-deploy/ directory.

Create the SSH key pair:
ssh-keygen -q -P '' -t rsa -b 4096 -m PEM -f domino.pem && chmod 600 domino.pem mv domino.pem* terraform/infra/
The key files sit alongside the infra Terraform configuration, which references them through the ssh_pvt_key_path variable.

Configure the `.tfvars` files

The three files at terraform/infra.tfvars, terraform/cluster.tfvars, and terraform/nodes.tfvars ship as scaffolding with null values. Populate the fields below and leave the rest as they are.

The availability zone IDs in these examples (usw2-az1, usw2-az2) are for us-west-2. Look up the IDs for your region:

aws ec2 describe-availability-zones --region <your-region> \
  --query 'AvailabilityZones[*].{Name:ZoneName,ID:ZoneId}'

deploy_id        = "domino-dp1"                   # Unique ID (alphanumeric + hyphens, max 15 chars)
region           = "us-west-2"
ssh_pvt_key_path = "terraform/infra/domino.pem"   # Moved here in the previous step

tags = {                                          # Optional; add if your organization requires them
  "owner"       = "domino"
  "environment" = "dataplane"
}

# Node groups: set the same availability_zone_ids values in nodes.tfvars
default_node_groups = {
  compute  = { availability_zone_ids = ["usw2-az1"] }
  gpu      = { availability_zone_ids = ["usw2-az2"] }
  platform = { availability_zone_ids = ["usw2-az2"] }
}

# Bastion: EC2 jump host for SSH tunnelling to the private EKS API endpoint
bastion = {
  enabled                  = true
  instance_type            = "t3.micro"
  authorized_ssh_ip_ranges = ["<YOUR_CORPORATE_CIDR>"]
  install_binaries         = true
  ami_id                   = null
  username                 = null
}

# Storage: the Data Plane does not own S3 or ECR; the control plane does
storage = {
  costs_enabled   = false
  filesystem_type = "efs"
  s3  = { create = false, force_destroy_on_deletion = true }
  ecr = { create = false, force_destroy_on_deletion = true }
}

kms = { enabled = true }

eks = {
  k8s_version = "<version from the Kubernetes compatibility table>"
  # Leave all other fields as-is; only update the block below:
  public_access = {
    enabled = false
    cidrs   = null
  }
}

default_node_groups = {
  compute  = { availability_zone_ids = ["usw2-az1"] }
  gpu      = { availability_zone_ids = ["usw2-az2"] }
  platform = { availability_zone_ids = ["usw2-az2"] }
}

additional_node_groups = null
karpenter_node_groups  = null
use_fips_endpoint      = false

Set k8s_version to a version listed for the Nexus data plane row of the Kubernetes compatibility table for your Domino release. Disabling public_access in cluster.tfvars gives you a private cluster, which is why the bastion and the SSH tunnel below exist.

Apply the stages

Initialize and plan all three stages:
./tf.sh all init ./tf.sh all plan
Review the plan output, then apply:
./tf.sh all apply
The infra stage takes roughly 10 minutes, cluster roughly 15, and nodes roughly 10.

Reach the cluster through the bastion SSH tunnel

The EKS API endpoint is private. During the cluster stage, Terraform generates a helper script at terraform/cluster/k8s-functions.sh, pre-filled with your bastion IP, kubeconfig path, tunnel port, and SSH key path. It handles the SSH SOCKS5 tunnel and the kubeconfig setup, so you do not edit it. Run these commands in order from domino-deploy/ in every new terminal session:

aws sts get-caller-identity              # Confirm AWS credentials are valid
source terraform/cluster/k8s-functions.sh
check_kubeconfig                         # Creates a proxy-aware kubeconfig copy
open_ssh_tunnel_to_k8s_api               # Opens the SSH SOCKS5 tunnel
kubectl get nodes                        # All nodes should report Ready

Close the tunnel when you finish:

close_ssh_tunnel_to_k8s_api

Keep this tunnel, or equivalent connectivity to the API endpoint, open while you run the bootstrap script during Data Plane registration. A private cluster is unreachable without it.

Install the Data Plane services

Set environment variables

Set the registry credentials and the agent version the installation uses:

unset HISTFILE   # Keeps credentials out of shell history
export QUAY_USERNAME=<your quay.io username>
export QUAY_PASSWORD=<your quay.io password>
export FLEETCOMMAND_AGENT_TAG=<tag matching the control plane>

Get the tag for your target release from the releases page, or from your Domino account team.

Install the Platform Operator

Point your Kubernetes context at the Data Plane cluster, then run:

ddlctl bootstrap --skip-domino-resource

Confirm the installation when prompted. This creates the domino-operator namespace and installs the Platform Operator Helm chart, without creating the Domino custom resource. Wait until the pods in domino-operator are running before you continue.

Create the configuration file

Generate a configuration file from the Data Plane preset:

ddlctl create config \
  --agent-version $FLEETCOMMAND_AGENT_TAG \
  --preset data-plane

On EKS, use --preset data-plane-eks instead. The preset sets control_plane: false and data_plane.type: remote, and omits email_notifications, telemetry, blob_storage, internal_docker_registry, external_docker_registry, and monitoring. See Data Plane deployments.If Terraform provisioned the cluster, pull its outputs into the configuration file rather than transcribing them:

./tf.sh infra output domino_config_values

Set the required fields

Edit the generated configuration file and set these values:

Field	Description	Example
`name`	Short name for the Data Plane, used as the subdomain	`test-dp`
`hostname`	Full hostname of your control plane	`domino.example.com`
`helm.image_registries[0].server`	The registry host	`quay.io`
`helm.image_registries[0].username`	The `quay.io` username	Value of `$QUAY_USERNAME`
`helm.image_registries[0].password`	The `quay.io` password	Value of `$QUAY_PASSWORD`

Search the generated file for < to find every placeholder that needs your value.

The Data Plane agent and the log forwarder each need to reach the control plane’s RabbitMQ and Vault services. Your Domino account team provides both hostnames:

release_overrides:
  data-plane-agent:
    chart_values:
      dataPlane:
        rabbitmq:
          host: <RabbitMQ host provided by Domino>
        vault:
          host: <Vault host provided by Domino>
  fluentd:
    chart_values:
      global:
        dataPlane:
          rabbitmq:
            host: <RabbitMQ host provided by Domino>

Configure ingress, TLS, and storage

Ingress configuration goes under release_overrides. Storage classes, the metrics server, and the autoscaler are top-level keys, siblings of release_overrides. Both differ by cloud provider.

storage_classes is optional on a Data Plane install. Configure it when the Data Plane needs its own block or shared storage, for example to serve Datasets.

AWS (EKS)
Azure (AKS)
GCP (GKE)

Attach the certificate to the load balancer with the aws-load-balancer-ssl-cert annotation. If you manage certificates in AWS Certificate Manager (ACM), request a certificate for the Data Plane hostname, give the DNS verification records to your Domino account team to confirm the domain, then use the resulting Amazon Resource Name (ARN):

release_overrides:
  nginx-ingress:
    chart_values:
      controller:
        kind: Deployment
        hostNetwork: false
        service:
          enabled: true
          externalTrafficPolicy: "Local"
          type: LoadBalancer
          annotations:
            service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: 'ELBSecurityPolicy-TLS-1-2-2017-01'
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: '<your ACM certificate ARN>'
            service.beta.kubernetes.io/aws-load-balancer-type: 'nlb'
            service.beta.kubernetes.io/aws-load-balancer-internal: 'false'
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: 'ssl'
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443'
            service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
            service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
            external-dns.alpha.kubernetes.io/hostname: <dp name>.<control plane domain>
          loadBalancerSourceRanges:
            - 0.0.0.0/0   # always 0.0.0.0/0; networkPolicy does the enforcement
        config:
          use-forwarded-headers: 'false'
          ssl-ciphers: 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:AES128-GCM-SHA256:AES128-SHA256:AES256-GCM-SHA384:AES256-SHA256:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA'
          ssl-protocols: 'TLSv1.2 TLSv1.3'
      networkPolicy:
        ipCidrs:
        - 10.0.0.0/16     # node network, adjust for your VPC
        - 100.64.0.0/16   # pod network, adjust for your VPC

The load balancer is deliberately open at loadBalancerSourceRanges, because networkPolicy.ipCidrs is what restricts access. Adjust those CIDRs to match your VPC. EKS also needs the metrics server, and takes the cluster autoscaler:

metrics_server:
  install: true

autoscaler:
  cloud_provider: aws
  auto_discovery:
    cluster_name: <Data Plane cluster name>
  aws:
    region: <AWS region>

Storage classes use Elastic Block Store (EBS) for block storage and Elastic File System (EFS) for the shared storage that Datasets need:

storage_classes:
  block:
    create: true
    name: dominodisk
    type: ebs
    access_modes:
    - ReadWriteOnce
    base_path: ''
    parameters:
      kmsKeyId: arn:aws:kms:<AWS region>:<AWS account ID>:key/<KMS key ID>
  shared:
    create: true
    type: efs
    name: dominoshared
    access_modes:
    - ReadWriteMany
    volume_capacity: 5Ti
    efs:
      region: <AWS region>
      filesystem_id: <EFS filesystem ID>
      access_point_id: <EFS access point ID>

The kmsKeyId parameter is optional. Include it when you require encryption with a customer-managed KMS key.If you run external-dns with IRSA, annotate its service account:

release_overrides:
  external-dns:
    chart_values:
      serviceAccount:
        create: true
        annotations:
          eks.amazonaws.com/role-arn: arn:aws:iam::<account ID>:role/<external-dns role name>
          eks.amazonaws.com/sts-regional-endpoints: 'true'

On Azure, the connectivity in Prerequisites is from the Data Plane’s virtual network to the control plane.Create a Kubernetes secret holding your TLS certificate, then reference it from the ingress configuration:

kubectl create secret tls dataplane-tls \
  -n domino-compute \
  --cert=<path to cert file> \
  --key=<path to key file>

release_overrides:
  nginx-ingress:
    chart_values:
      controller:
        kind: Deployment
        hostNetwork: false
        service:
          enabled: true
          type: LoadBalancer
        extraArgs:
          default-ssl-certificate: domino-compute/dataplane-tls

After the deployment finishes, retrieve the load balancer’s external IP and create a DNS record pointing the Data Plane hostname at it:

kubectl get svc -n domino-compute nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Storage configuration on AKS depends on which storage solution you choose: Azure Files for shared storage, Azure Managed Disks for block storage, or NetApp Cloud Volumes ONTAP for enterprise storage requirements. Contact your Domino account team for guidance on configuring them.

Create a Kubernetes secret holding your TLS certificate before you deploy:

kubectl create secret tls <TLS secret name> \
  -n <namespace> \
  --cert=<path to cert file> \
  --key=<path to key file>

Reference that secret from the ingress configuration, replacing <namespace> with the namespace holding the secret and <TLS secret name> with its name:

release_overrides:
  nginx-ingress:
    chart_values:
      controller:
        hostNetwork: false
        kind: Deployment
        service:
          enabled: true
          type: LoadBalancer
          annotations:
            cloud.google.com/backend-config: '{"ports": {"80":"nginx-ingress-controller","443":"nginx-ingress-controller"}}'
        extraArgs:
          default-ssl-certificate: <namespace>/<TLS secret name>

Storage classes use a Google Compute Engine (GCE) persistent disk for block storage and Network File System (NFS) for the shared storage that Datasets need. A Cloud Filestore instance IP works as the NFS server:

storage_classes:
  block:
    create: true
    name: dominodisk
    type: gce
    access_modes:
    - ReadWriteOnce
    base_path: ''
  shared:
    create: true
    name: dominoshared
    type: nfs
    access_modes:
    - ReadWriteMany
    volume_capacity: 2.5Ti
    nfs:
      server: <NFS server IP>
      mount_path: <export path>
      mount_options: []

Add optional configuration

Add any of these to the generated configuration file if your deployment needs them:

GPU support

gpu:
  enabled: true

Node selectors

global_node_selectors:
  dominodatalab.com/domino-node: 'true'

Istio

istio:
  enabled: true
  install: true
  cni: false

Setting istio.enabled here is not sufficient on its own. Also select Enable Istio when you register the Data Plane.

Monitoring

monitoring:
  prometheus_metrics: true
  gpu_metrics: true
  newrelic:
    apm: false
    infrastructure: false
    license_key: ''

Datasets

Datasets on a remote Data Plane need the volume operator and the filetask operator:

release_overrides:
  data-plane:
    chart_values:
      enable_volume_operator: true
      enable_filetask_operator: true

Apply the configuration

Create the Domino custom resource, which the Platform Operator reconciles into the running Data Plane:

ddlctl create domino \
  --agent-version $FLEETCOMMAND_AGENT_TAG \
  --config ./<generated configuration file>

Watch progress with kubectl get pods -A -w.

Point DNS at the load balancer

If your ingress configuration set external-dns annotations, DNS is configured for you. Otherwise, retrieve the load balancer address and create the record yourself:

kubectl get svc -n domino-platform -l app=nginx-ingress

On AWS, create a canonical name (CNAME) record pointing at the load balancer address, not an address (A) record.

On a Domino Cloud deployment, give the load balancer IP to your Domino account team, who configure DNS for you. Confirm TLS works by opening https:// plus your Data Plane hostname in a browser.

Verify the installation

Confirm no pod is stuck. This command lists only pods that are neither running nor completed, so an empty result means the cluster is healthy:
kubectl get po -A | grep -v Running | grep -v Completed
To read the detail instead, list the two Domino namespaces directly:
kubectl get pods -n domino-platform kubectl get pods -n domino-compute
Check the data-plane-agent pod logs. Seeing this message means the install succeeded and the agent is waiting for the bootstrap that registration provides:
Secret agent-bootstrap not found; waiting...

Register the Data Plane

The Data Plane is installed but not yet connected to anything. Register it with the control plane to create its record in the admin portal, generate a bootstrap token, and run the bootstrap script against the cluster you just built. Registration is what turns Secret agent-bootstrap not found; waiting... into a healthy Data Plane.

Architecture & deployment

Domino installation

Identity & access

Infrastructure & compute

Platform configuration

Data administration

Operations, monitoring & cost

Security & compliance

Troubleshooting

Install a Data Plane

Prerequisites

Cluster node pools

Provision the cluster

Sample AKS node pool configuration

Provision an EKS cluster with Terraform

Bootstrap the deployment directory

Configure the `.tfvars` files

Apply the stages

Reach the cluster through the bastion SSH tunnel

Install the Data Plane services

Verify the installation

Register the Data Plane

​Prerequisites

​Cluster node pools

​Provision the cluster

​Sample AKS node pool configuration

​Provision an EKS cluster with Terraform

​Bootstrap the deployment directory

​Configure the .tfvars files

​Apply the stages

​Reach the cluster through the bastion SSH tunnel

​Install the Data Plane services

​Verify the installation

​Register the Data Plane

​Related

Prerequisites

Cluster node pools

Provision the cluster

Sample AKS node pool configuration

Provision an EKS cluster with Terraform

Bootstrap the deployment directory

Configure the `.tfvars` files

Apply the stages

Reach the cluster through the bastion SSH tunnel

Install the Data Plane services

Verify the installation

Register the Data Plane

Related