How to Troubleshoot Kubernetes Autoscaling (ASG) Issues in Domino

Version/Environment (if relevant):

All versions of Domino; applies to AWS-based deployments using Kubernetes Cluster Autoscaler and AutoScaling Groups (ASGs)

Issue:

Executions fail to start or pods fail to schedule due to autoscaling issues. Common symptoms include pods stuck in Pending state, scale-up not triggering, or volume node affinity conflicts. Errors appear in the support bundle’s execution.log or events.json, for example:

2023-06-26T15:06:39.204Z: pod didn't trigger scale-up:2node(s) had volume node affinity conflict,26node(s) didn't match Pod's node affinity/selector
Please check if the hardware tier'p3-Large-GPU'has a valid nodePool

"resourceKind":"Pod",
"reason":"NotTriggerScaleUp",
"description":"pod didn't trigger scale-up: 26 node(s) didn't match Pod's node affinity/selector, 2 node(s) had volume node affinity conflict",

Root Cause:

Autoscaling failures in Domino are typically caused by one or more of the following:

Autoscaling is not enabled in the cluster yaml
Pod resource requests (CPU/memory) exceed what any node in the group can provide
Mismatch between ASG configuration in AWS and the Kubernetes autoscaler deployment settings
ASG configured across multiple Availability Zones, causing volume node affinity conflicts
Node labels not matching the Hardware Tier’s node selector or affinity rules
ASG max node group size has been reached

Resolution:

1. Verify autoscaling is enabled in the cluster yaml:

autoscaler:
 enabled: true

2. Describe the unscheduled pod and inspect its Events section: Common event messages and their meanings:

Autoscaler intentionally not triggering scale-up — pods request more CPU/memory than any node can provide:

Normal NotTriggerScaleUp 2m24s (x18 over 27m) cluster-autoscaler pod didn't trigger scale-up (it wouldn't fit if a new node is added): 45 Insufficient memory, 3 max node group size reached, 45 Insufficient cpu

Autoscaler not triggering when it should — mismatch between AWS ASGs and Kubernetes settings:

Warning FailedScheduling <unknown> default-scheduler 0/12 nodes are available: 1 Insufficient memory, 1 node(s) didn't match node selector, 10 Insufficient cpu.

Volume node affinity conflict — ASG spans multiple AZs that don’t match storage volume AZs:

pod didn't trigger scale-up (it wouldn't fit if a new node is added): 1 node(s) had volume node affinity conflict

3. Verify and reconcile ASG settings between AWS and Kubernetes:

kubectl get configmap cluster-autoscaler-status -n <install namespace> -o yaml
kubectl edit deployment -n <platform namespace> <cluster autoscaler deployment name>

Ensure each ASG has only one Availability Zone, both in the AWS console and in the configmap. Storage volume AZs must match ASG instance AZs. 4. If scale-up is intentionally not triggering due to insufficient resources, reduce pod resource requests via Hardware Tier settings. Refer to: https://admin.dominodatalab.com/en/4.4.1/compute/hardware-tiers.html?highlight=hardware%20tier 5. Review min, max, and currently running nodes:

kubectl describe cm -n <could be platform namespace, could be something like kube-system> cluster-autoscaler-status

Notes/Information:

Other less common scheduling messages and their meanings:

2 node(s) didn't match pod affinity rules, 2 node(s) didn't match pod affinity/anti-affinity rules — Pod labels or affinity rules don’t match the available nodes. Describe the pod and check the affinity section.
11 node(s) didn't match Pod's node affinity or 4 node(s) didn't match node selector — Nodes exist but don’t carry the label required by the Hardware Tier’s Node Selector (e.g., dominodatalab.com/node-pool=xlarge-memory). Check pod affinity, Node Selector, and node labels.
3 max node group size reached — The ASG’s maximum node count in AWS has been reached. Increase the ASG maxSize to allow additional nodes to spin up.

If the Kubernetes autoscaler deployment is edited directly, also update the autoscaler section of the cluster yaml (deploy manifest) to persist changes across upgrades and maintenance.

​Version/Environment (if relevant):

​Issue:

​Root Cause:

​Resolution:

​Notes/Information:

Version/Environment (if relevant):

Issue:

Root Cause:

Resolution:

Notes/Information: