vm_memory_high_watermark" error in RabbitMQ service logs

Issue:

Examining the logs for the RabbitMQ StatefulSet/rabbitmq-ha-XXX shows the following error:

[info] <0.449.0> vm_memory_high_watermark set. Memory used:1858117632 allowed:1717986918
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.
[warning] <0.447.0>
[warning] <0.447.0> **********************************************************
[warning] <0.447.0> *** Publishers will be blocked until this alarm clears ***
[warning] <0.447.0> **********************************************************
[warning] <0.447.0>
[warning] <0.447.0> memory resource limit alarm cleared on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'
[warning] <0.447.0> memory resource limit alarm cleared across the cluster
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.

This condition typically surfaces in the Domino frontend as an error similar to:

“Workspace could not be started. Data plane Local is not in a healthy state: DataPlaneStatus(Disconnected, Not responding, Some(1700470196563))“

Root Cause:

When RabbitMQ reaches the vm_memory_high_watermark threshold, it sets a memory resource limit alarm on the affected node and stops accepting new connections. The data-plane-agent service detects its connections to RabbitMQ are refused and marks itself as unhealthy, which causes the frontend error above. To confirm this in the logs, retrieve the RabbitMQ container logs using one of the following methods. Recent versions of Domino — stream logs directly from the Kubernetes container:

kubectl logs -n <your-domino-platform-namespace> rabbitmq-ha-311-0 --follow

Note: The pod name rabbitmq-ha-311-0 is illustrative. Replace it with the name of the affected pod in your environment.

Older versions of Domino — exec into the container and read the log file:

kubectl exec -it -n <your-domino-platform-namespace> rabbitmq-ha-311-0 -c rabbitmq-ha -- /bin/bash
cat /var/log/rabbitmq/*log

Look for vm_memory_high_watermark set and memory resource limit alarm set entries to confirm the root cause.

Resolution:

Temporary fix Restart the affected RabbitMQ pod to clear the alarm and restore operations. Use the following command, replacing the pod name with the affected pod in your environment:

kubectl delete pod -n <your-domino-platform-namespace> rabbitmq-ha-311-0

The pod restarts automatically. However, memory usage typically increases over time and the alarm recurs without a permanent configuration change. Permanent fix Increase the vm_memory_high_watermark setting in the rabbitmq-ha ConfigMap to set an absolute memory limit appropriate for your container’s resource limits. The default value is vm_memory_high_watermark.relative = 0.4, which allocates 40% of available memory. This default protects against out-of-memory (OOM) situations. Where container memory limits allow, replacing the relative value with an explicit absolute value provides more predictable behavior. For example, on a container with a 6 GB memory limit, 3500MB is an acceptable threshold. Step 1: Edit the ConfigMap.

kubectl edit cm -n <your-domino-platform-namespace> rabbitmq-ha

Step 2: Replace the relative setting with an absolute value.

## Memory Threshold
# vm_memory_high_watermark.relative = 0.4
vm_memory_high_watermark.absolute = 3500MB

Step 3: Restart the RabbitMQ pods to apply the change. Run the following command to perform a rolling restart of the StatefulSet:

kubectl rollout restart statefulset -n <your-domino-platform-namespace> rabbitmq-ha

Step 4: Verify the new watermark is active. After the pods restart, check the logs for confirmation that the updated value loaded:

kubectl logs -n <your-domino-platform-namespace> <rabbitmq-pod-name> | grep vm_memory_high_watermark

Confirm the output references the absolute value you set (for example, 3500MB) rather than the previous relative value.

​Issue:

​Root Cause:

​Resolution:

Issue:

Root Cause:

Resolution: