> ## Documentation Index > Fetch the complete documentation index at: https://docs.domino.ai/llms.txt > Use this file to discover all available pages before exploring further. # Install a Data Plane > Provision a Kubernetes cluster and install the Domino Data Plane services on it with ddlctl, on Amazon EKS, Azure AKS, or Google GKE. Installing a Data Plane is the first half of bringing one up. You provision a Kubernetes cluster, install the Domino services onto it, and verify that the Data Plane agent is waiting for its bootstrap. The second half, [registering the Data Plane](/6.3/admin/platform-configuration/manage-data-planes#register-a-data-plane), connects it to the control plane and is where this page ends. Starting with Domino 6.1, Data Planes are installed with the [Platform Operator](/6.3/admin/installation/platform-operator/overview), the same deployment model the control plane uses. That gives both planes one configuration format and one upgrade path. Set the `FLEETCOMMAND_AGENT_TAG` and the Data Plane configuration correctly before you apply them. Incorrect values cannot be fixed or updated once the install completes: the deployment is left partially broken and recovers only through manual intervention. ## Prerequisites * A functional control plane running Domino 6.1 or later, with Nexus enabled. * A Kubernetes cluster for the Data Plane. See [Provision the cluster](#provision-the-cluster). * Network connectivity from the Data Plane cluster to four control plane endpoints: | Endpoint | Example | Purpose | | :------------------------------- | :---------------------------- | :-------------------- | | Control plane container registry | `registry.domino.example.com` | Pull container images | | Control plane hostname | `domino.example.com` | API communication | | RabbitMQ service | `rabbit.domino.example.com` | Message queue | | Vault service | `vault.domino.example.com` | Secrets management | * [Configuration records](/6.3/admin/configuration/central-configuration#data-planes) for the RabbitMQ, Vault, and Docker registry hosts, plus any custom ports. * A Domain Name System (DNS) subdomain of your control plane domain. If the control plane is `example.com`, the Data Plane is something like `dp.example.com`. * A valid Transport Layer Security (TLS) certificate for the Data Plane hostname. * Registry credentials for `quay.io`, provided by your Domino account team. Install these tools on the workstation you run the installation from: | Tool | Purpose | | :-------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------- | | `ddlctl` | The Domino deployer command line. See [Install ddlctl](/6.3/admin/installation/platform-operator/ddlctl/install). | | `kubectl` | Applies and inspects Kubernetes resources on the Data Plane cluster. | | Docker | Certain `ddlctl` operations require access to the Docker Engine API. See [Docker engine](/6.3/admin/installation/platform-operator/ddlctl/docker-engine). | | `curl` | Downloads the bootstrap script during registration. | ### Cluster node pools The Data Plane cluster needs at least two node pools, labeled so Domino can schedule against them: | Node pool | Label | Runs | Scaling | | :-------- | :-------------------------------------- | :------------------ | :------ | | Platform | `dominodatalab.com/node-pool: platform` | Data Plane services | 0-1 | | Compute | `dominodatalab.com/node-pool: default` | User workloads | 0-n | Size the nodes for your cloud provider: | Cloud provider | Platform pool | Compute pool | | :-------------------------------------- | :------------------------- | :---------------------- | | Amazon Elastic Kubernetes Service (EKS) | `m5.2xlarge` or similar | Based on workload needs | | Azure Kubernetes Service (AKS) | `Standard_D8s_v5` | `Standard_D16s_v5` | | Google Kubernetes Engine (GKE) | `n1-standard-8` or similar | Based on workload needs | On AKS, spread the compute pool across all 3 availability zones by creating one pool per zone. The platform pool needs no zonal distribution, because it holds no persistent storage. The Data Plane cluster's Kubernetes version can differ from the control plane's, as long as it sits in the [compatible range](/6.3/admin/architecture-and-deployment/kubernetes-infrastructure#compatibility-matrix) for your Domino version. ## Provision the cluster Provision an EKS, AKS, or GKE cluster that meets the [node pool requirements](#cluster-node-pools) above, then point your default Kubernetes context at it. For EKS, Domino publishes a Terraform module that provisions the whole stack. One difference from a control plane deployment is worth calling out before you start: a Data Plane does not own the S3 bucket or the Elastic Container Registry (ECR), which stay with the control plane. [Deploy Domino on AWS EKS](/6.3/admin/installation/manage-deployments/aws/deploy) covers the same `data-plane-eks` preset from the control plane installer's perspective, including the Terraform values to pull into the configuration file. ### Sample AKS node pool configuration Create one platform pool and one compute pool per availability zone. For the compute pools, name them `compute1`, `compute2`, and `compute3`, and set `availabilityZones` to `["1"]`, `["2"]`, and `["3"]` respectively. ```json Platform node pool theme={null} { "name": "platform", "vmSize": "Standard_D8s_v5", "osDiskSizeGB": 128, "osDiskType": "Ephemeral", "kubeletDiskType": "OS", "maxPods": 110, "type": "VirtualMachineScaleSets", "maxCount": 1, "minCount": 0, "enableAutoScaling": true, "scaleDownMode": "Delete", "enableNodePublicIP": false, "nodeLabels": { "dominodatalab.com/node-pool": "platform" }, "mode": "User", "osType": "Linux", "osSKU": "Ubuntu" } ``` ```json Compute node pool theme={null} { "name": "compute1", "vmSize": "Standard_D16s_v3", "osDiskSizeGB": 128, "osDiskType": "Ephemeral", "kubeletDiskType": "OS", "maxPods": 110, "type": "VirtualMachineScaleSets", "maxCount": 3, "minCount": 0, "enableAutoScaling": true, "scaleDownMode": "Delete", "availabilityZones": ["1"], "enableNodePublicIP": false, "nodeLabels": { "dominodatalab.com/node-pool": "default" }, "mode": "User", "osType": "Linux", "osSKU": "Ubuntu" } ``` ### Provision an EKS cluster with Terraform The [terraform-aws-eks](https://github.com/dominodatalab/terraform-aws-eks) module uses a three-stage deployment driven by a `tf.sh` wrapper script. Apply the stages in order, because each depends on the outputs of the one before it: | Stage | What it creates | `.tfvars` file | | :-------- | :---------------------------------------------------------------------------------------------------------------------------------- | :------------------------- | | `infra` | Virtual Private Cloud (VPC), subnets, Key Management Service (KMS) key, bastion, Secure Shell (SSH) key pair | `terraform/infra.tfvars` | | `cluster` | EKS control plane, Calico Container Network Interface (CNI), Identity and Access Management (IAM) Roles for Service Accounts (IRSA) | `terraform/cluster.tfvars` | | `nodes` | EKS-managed node groups | `terraform/nodes.tfvars` | Beyond the tools listed under [Prerequisites](#prerequisites), Terraform provisioning needs [Terraform](https://developer.hashicorp.com/terraform/install) 1.3 or later, the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) v2, [Helm](https://helm.sh/docs/intro/install/) 3.x, and [hcledit](https://github.com/minamijoyo/hcledit) on your `PATH`. Configure AWS credentials with permission to create VPC resources, EKS clusters, EC2 instances, IAM roles, and security groups, then confirm the right account is active: ```shell theme={null} aws sts get-caller-identity ``` The command returns your `UserId`, `Account`, and `Arn`. #### Bootstrap the deployment directory Rather than cloning the full repository, use `terraform init -from-module` to pull the `examples/deploy` scaffold at a specific version tag into a local directory. That also runs `set-mod-version.sh`, which pins all three module sources to the same tag. 1. Install `hcledit`, which `set-mod-version.sh` requires: ```shell macOS theme={null} brew install minamijoyo/hcledit/hcledit ``` ```shell Go theme={null} go install github.com/minamijoyo/hcledit@latest ``` 2. Set `MOD_VERSION` to the module version validated for your Domino release, listed on [Provision Terraform infrastructure for AWS](/6.3/admin/installation/manage-deployments/aws/infrastructure), then bootstrap the directory: ```shell theme={null} export MOD_VERSION='' export DEPLOY_DIR='domino-deploy' mkdir -p "$DEPLOY_DIR" terraform -chdir="$DEPLOY_DIR" init -backend=false \ -from-module="github.com/dominodatalab/terraform-aws-eks.git//examples/deploy?ref=${MOD_VERSION}" cd $DEPLOY_DIR unset DEPLOY_DIR ./set-mod-version.sh "$MOD_VERSION" ``` Run every later `./tf.sh` command from this `domino-deploy/` directory. 3. Create the SSH key pair: ```shell theme={null} ssh-keygen -q -P '' -t rsa -b 4096 -m PEM -f domino.pem && chmod 600 domino.pem mv domino.pem* terraform/infra/ ``` The key files sit alongside the `infra` Terraform configuration, which references them through the `ssh_pvt_key_path` variable. #### Configure the `.tfvars` files The three files at `terraform/infra.tfvars`, `terraform/cluster.tfvars`, and `terraform/nodes.tfvars` ship as scaffolding with null values. Populate the fields below and leave the rest as they are. The availability zone IDs in these examples (`usw2-az1`, `usw2-az2`) are for `us-west-2`. Look up the IDs for your region: ```shell theme={null} aws ec2 describe-availability-zones --region \ --query 'AvailabilityZones[*].{Name:ZoneName,ID:ZoneId}' ``` ```hcl terraform/infra.tfvars theme={null} deploy_id = "domino-dp1" # Unique ID (alphanumeric + hyphens, max 15 chars) region = "us-west-2" ssh_pvt_key_path = "terraform/infra/domino.pem" # Moved here in the previous step tags = { # Optional; add if your organization requires them "owner" = "domino" "environment" = "dataplane" } # Node groups: set the same availability_zone_ids values in nodes.tfvars default_node_groups = { compute = { availability_zone_ids = ["usw2-az1"] } gpu = { availability_zone_ids = ["usw2-az2"] } platform = { availability_zone_ids = ["usw2-az2"] } } # Bastion: EC2 jump host for SSH tunnelling to the private EKS API endpoint bastion = { enabled = true instance_type = "t3.micro" authorized_ssh_ip_ranges = [""] install_binaries = true ami_id = null username = null } # Storage: the Data Plane does not own S3 or ECR; the control plane does storage = { costs_enabled = false filesystem_type = "efs" s3 = { create = false, force_destroy_on_deletion = true } ecr = { create = false, force_destroy_on_deletion = true } } kms = { enabled = true } ``` ```hcl terraform/cluster.tfvars theme={null} eks = { k8s_version = "" # Leave all other fields as-is; only update the block below: public_access = { enabled = false cidrs = null } } ``` ```hcl terraform/nodes.tfvars theme={null} default_node_groups = { compute = { availability_zone_ids = ["usw2-az1"] } gpu = { availability_zone_ids = ["usw2-az2"] } platform = { availability_zone_ids = ["usw2-az2"] } } additional_node_groups = null karpenter_node_groups = null use_fips_endpoint = false ``` Set `k8s_version` to a version listed for the `Nexus data plane` row of the [Kubernetes compatibility table](/6.3/admin/architecture-and-deployment/kubernetes-infrastructure#compatibility-matrix) for your Domino release. Disabling `public_access` in `cluster.tfvars` gives you a private cluster, which is why the bastion and the SSH tunnel below exist. #### Apply the stages 1. Initialize and plan all three stages: ```shell theme={null} ./tf.sh all init ./tf.sh all plan ``` 2. Review the plan output, then apply: ```shell theme={null} ./tf.sh all apply ``` The `infra` stage takes roughly 10 minutes, `cluster` roughly 15, and `nodes` roughly 10. #### Reach the cluster through the bastion SSH tunnel The EKS API endpoint is private. During the `cluster` stage, Terraform generates a helper script at `terraform/cluster/k8s-functions.sh`, pre-filled with your bastion IP, kubeconfig path, tunnel port, and SSH key path. It handles the SSH SOCKS5 tunnel and the kubeconfig setup, so you do not edit it. Run these commands in order from `domino-deploy/` in every new terminal session: ```shell theme={null} aws sts get-caller-identity # Confirm AWS credentials are valid source terraform/cluster/k8s-functions.sh check_kubeconfig # Creates a proxy-aware kubeconfig copy open_ssh_tunnel_to_k8s_api # Opens the SSH SOCKS5 tunnel kubectl get nodes # All nodes should report Ready ``` Close the tunnel when you finish: ```shell theme={null} close_ssh_tunnel_to_k8s_api ``` Keep this tunnel, or equivalent connectivity to the API endpoint, open while you run the bootstrap script during [Data Plane registration](/6.3/admin/platform-configuration/manage-data-planes#register-a-data-plane). A private cluster is unreachable without it. ## Install the Data Plane services Set the registry credentials and the agent version the installation uses: ```shell theme={null} unset HISTFILE # Keeps credentials out of shell history export QUAY_USERNAME= export QUAY_PASSWORD= export FLEETCOMMAND_AGENT_TAG= ``` Get the tag for your target release from the [releases](/6.3/admin/installation/releases) page, or from your Domino account team. Point your Kubernetes context at the Data Plane cluster, then run: ```shell theme={null} ddlctl bootstrap --skip-domino-resource ``` Confirm the installation when prompted. This creates the `domino-operator` namespace and installs the Platform Operator Helm chart, without creating the Domino custom resource. Wait until the pods in `domino-operator` are running before you continue. Generate a configuration file from the Data Plane preset: ```shell theme={null} ddlctl create config \ --agent-version $FLEETCOMMAND_AGENT_TAG \ --preset data-plane ``` On EKS, use `--preset data-plane-eks` instead. The preset sets `control_plane: false` and `data_plane.type: remote`, and omits `email_notifications`, `telemetry`, `blob_storage`, `internal_docker_registry`, `external_docker_registry`, and `monitoring`. See [Data Plane deployments](/6.3/admin/installation/manage-deployments#data-plane-deployments). If Terraform provisioned the cluster, pull its outputs into the configuration file rather than transcribing them: ```shell theme={null} ./tf.sh infra output domino_config_values ``` Edit the generated configuration file and set these values: | Field | Description | Example | | :---------------------------------- | :--------------------------------------------------- | :------------------------ | | `name` | Short name for the Data Plane, used as the subdomain | `test-dp` | | `hostname` | Full hostname of your control plane | `domino.example.com` | | `helm.image_registries[0].server` | The registry host | `quay.io` | | `helm.image_registries[0].username` | The `quay.io` username | Value of `$QUAY_USERNAME` | | `helm.image_registries[0].password` | The `quay.io` password | Value of `$QUAY_PASSWORD` | Search the generated file for `<` to find every placeholder that needs your value. The Data Plane agent and the log forwarder each need to reach the control plane's RabbitMQ and Vault services. Your Domino account team provides both hostnames: ```yaml theme={null} release_overrides: data-plane-agent: chart_values: dataPlane: rabbitmq: host: vault: host: fluentd: chart_values: global: dataPlane: rabbitmq: host: ``` Ingress configuration goes under `release_overrides`. Storage classes, the metrics server, and the autoscaler are top-level keys, siblings of `release_overrides`. Both differ by cloud provider. `storage_classes` is optional on a Data Plane install. Configure it when the Data Plane needs its own block or shared storage, for example to serve Datasets. Attach the certificate to the load balancer with the `aws-load-balancer-ssl-cert` annotation. If you manage certificates in [AWS Certificate Manager (ACM)](https://docs.aws.amazon.com/acm/latest/userguide/acm-overview.html), request a certificate for the Data Plane hostname, give the DNS verification records to your Domino account team to confirm the domain, then use the resulting Amazon Resource Name (ARN): ```yaml theme={null} release_overrides: nginx-ingress: chart_values: controller: kind: Deployment hostNetwork: false service: enabled: true externalTrafficPolicy: "Local" type: LoadBalancer annotations: service.beta.kubernetes.io/aws-load-balancer-ssl-negotiation-policy: 'ELBSecurityPolicy-TLS-1-2-2017-01' service.beta.kubernetes.io/aws-load-balancer-ssl-cert: '' service.beta.kubernetes.io/aws-load-balancer-type: 'nlb' service.beta.kubernetes.io/aws-load-balancer-internal: 'false' service.beta.kubernetes.io/aws-load-balancer-backend-protocol: 'ssl' service.beta.kubernetes.io/aws-load-balancer-ssl-ports: '443' service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600' service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true' external-dns.alpha.kubernetes.io/hostname: . loadBalancerSourceRanges: - 0.0.0.0/0 # always 0.0.0.0/0; networkPolicy does the enforcement config: use-forwarded-headers: 'false' ssl-ciphers: 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:AES128-GCM-SHA256:AES128-SHA256:AES256-GCM-SHA384:AES256-SHA256:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA' ssl-protocols: 'TLSv1.2 TLSv1.3' networkPolicy: ipCidrs: - 10.0.0.0/16 # node network, adjust for your VPC - 100.64.0.0/16 # pod network, adjust for your VPC ``` The load balancer is deliberately open at `loadBalancerSourceRanges`, because `networkPolicy.ipCidrs` is what restricts access. Adjust those CIDRs to match your VPC. EKS also needs the metrics server, and takes the cluster autoscaler: ```yaml theme={null} metrics_server: install: true autoscaler: cloud_provider: aws auto_discovery: cluster_name: aws: region: ``` Storage classes use Elastic Block Store (EBS) for block storage and Elastic File System (EFS) for the shared storage that Datasets need: ```yaml theme={null} storage_classes: block: create: true name: dominodisk type: ebs access_modes: - ReadWriteOnce base_path: '' parameters: kmsKeyId: arn:aws:kms:::key/ shared: create: true type: efs name: dominoshared access_modes: - ReadWriteMany volume_capacity: 5Ti efs: region: filesystem_id: access_point_id: ``` The `kmsKeyId` parameter is optional. Include it when you require encryption with a customer-managed KMS key. If you run `external-dns` with IRSA, annotate its service account: ```yaml theme={null} release_overrides: external-dns: chart_values: serviceAccount: create: true annotations: eks.amazonaws.com/role-arn: arn:aws:iam:::role/ eks.amazonaws.com/sts-regional-endpoints: 'true' ``` On Azure, the connectivity in [Prerequisites](#prerequisites) is from the Data Plane's virtual network to the control plane. Create a Kubernetes secret holding your TLS certificate, then reference it from the ingress configuration: ```shell theme={null} kubectl create secret tls dataplane-tls \ -n domino-compute \ --cert= \ --key= ``` ```yaml theme={null} release_overrides: nginx-ingress: chart_values: controller: kind: Deployment hostNetwork: false service: enabled: true type: LoadBalancer extraArgs: default-ssl-certificate: domino-compute/dataplane-tls ``` After the deployment finishes, retrieve the load balancer's external IP and create a DNS record pointing the Data Plane hostname at it: ```shell theme={null} kubectl get svc -n domino-compute nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` Storage configuration on AKS depends on which storage solution you choose: Azure Files for shared storage, Azure Managed Disks for block storage, or NetApp Cloud Volumes ONTAP for enterprise storage requirements. Contact your Domino account team for guidance on configuring them. Create a Kubernetes secret holding your TLS certificate before you deploy: ```shell theme={null} kubectl create secret tls \ -n \ --cert= \ --key= ``` Reference that secret from the ingress configuration, replacing `` with the namespace holding the secret and `` with its name: ```yaml theme={null} release_overrides: nginx-ingress: chart_values: controller: hostNetwork: false kind: Deployment service: enabled: true type: LoadBalancer annotations: cloud.google.com/backend-config: '{"ports": {"80":"nginx-ingress-controller","443":"nginx-ingress-controller"}}' extraArgs: default-ssl-certificate: / ``` Storage classes use a Google Compute Engine (GCE) persistent disk for block storage and Network File System (NFS) for the shared storage that Datasets need. A Cloud Filestore instance IP works as the NFS server: ```yaml theme={null} storage_classes: block: create: true name: dominodisk type: gce access_modes: - ReadWriteOnce base_path: '' shared: create: true name: dominoshared type: nfs access_modes: - ReadWriteMany volume_capacity: 2.5Ti nfs: server: mount_path: mount_options: [] ``` Add any of these to the generated configuration file if your deployment needs them: ```yaml theme={null} gpu: enabled: true ``` ```yaml theme={null} global_node_selectors: dominodatalab.com/domino-node: 'true' ``` ```yaml theme={null} istio: enabled: true install: true cni: false ``` Setting `istio.enabled` here is not sufficient on its own. Also select **Enable Istio** when you [register the Data Plane](/6.3/admin/platform-configuration/manage-data-planes#register-a-data-plane). ```yaml theme={null} monitoring: prometheus_metrics: true gpu_metrics: true newrelic: apm: false infrastructure: false license_key: '' ``` Datasets on a remote Data Plane need the volume operator and the filetask operator: ```yaml theme={null} release_overrides: data-plane: chart_values: enable_volume_operator: true enable_filetask_operator: true ``` Create the Domino custom resource, which the Platform Operator reconciles into the running Data Plane: ```shell theme={null} ddlctl create domino \ --agent-version $FLEETCOMMAND_AGENT_TAG \ --config ./ ``` Watch progress with `kubectl get pods -A -w`. If your ingress configuration set `external-dns` annotations, DNS is configured for you. Otherwise, retrieve the load balancer address and create the record yourself: ```shell theme={null} kubectl get svc -n domino-platform -l app=nginx-ingress ``` On AWS, create a canonical name (CNAME) record pointing at the load balancer address, not an address (A) record. On a Domino Cloud deployment, give the load balancer IP to your Domino account team, who configure DNS for you. Confirm TLS works by opening `https://` plus your Data Plane hostname in a browser. ## Verify the installation 1. Confirm no pod is stuck. This command lists only pods that are neither running nor completed, so an empty result means the cluster is healthy: ```shell theme={null} kubectl get po -A | grep -v Running | grep -v Completed ``` To read the detail instead, list the two Domino namespaces directly: ```shell theme={null} kubectl get pods -n domino-platform kubectl get pods -n domino-compute ``` 2. Check the `data-plane-agent` pod logs. Seeing this message means the install succeeded and the agent is waiting for the bootstrap that registration provides: ``` Secret agent-bootstrap not found; waiting... ``` ## Register the Data Plane The Data Plane is installed but not yet connected to anything. [Register it with the control plane](/6.3/admin/platform-configuration/manage-data-planes#register-a-data-plane) to create its record in the admin portal, generate a bootstrap token, and run the bootstrap script against the cluster you just built. Registration is what turns `Secret agent-bootstrap not found; waiting...` into a healthy Data Plane. ## Related * [Data Planes administration](/6.3/admin/platform-configuration/manage-data-planes) * [Domino Nexus hybrid architecture](/6.3/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture) * [Platform Operator overview](/6.3/admin/installation/platform-operator/overview) * [Kubernetes infrastructure](/6.3/admin/architecture-and-deployment/kubernetes-infrastructure) * [Deploy Domino on AWS EKS](/6.3/admin/installation/manage-deployments/aws/deploy) * [Provision Terraform infrastructure for AWS](/6.3/admin/installation/manage-deployments/aws/infrastructure) ## Related topics - [Install Domino](/6.3/admin/installation/manage-deployments/index.md) - [Data Planes administration](/6.3/admin/platform-configuration/manage-data-planes.md) - [Install configuration reference](/6.3/admin/configuration/installer-configuration.md) - [Remote Data Plane monitoring](/6.3/admin/operations/monitoring/remote-data-plane-monitoring.md)