> ## Documentation Index
> Fetch the complete documentation index at: https://docs.domino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# vm_memory_high_watermark" error in RabbitMQ service logs

> Resolve RabbitMQ vm_memory_high_watermark memory alarms in Domino on Kubernetes by restarting the affected pod or permanently raising the absolute memory threshold in the ConfigMap.

## **Issue:**

Examining the logs for the RabbitMQ StatefulSet/rabbitmq-ha-XXX shows the following error:

```
[info] <0.449.0> vm_memory_high_watermark set. Memory used:1858117632 allowed:1717986918
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.
[warning] <0.447.0>
[warning] <0.447.0> **********************************************************
[warning] <0.447.0> *** Publishers will be blocked until this alarm clears ***
[warning] <0.447.0> **********************************************************
[warning] <0.447.0>
[warning] <0.447.0> memory resource limit alarm cleared on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'
[warning] <0.447.0> memory resource limit alarm cleared across the cluster
[warning] <0.447.0> memory resource limit alarm set on node 'rabbit@rabbitmq-ha-311-0.rabbitmq-ha-discovery.example.svc.cluster.local'.
```

This condition typically surfaces in the Domino frontend as an error similar to:

> "Workspace could not be started. Data plane Local is not in a healthy state: DataPlaneStatus(Disconnected, Not responding, Some(1700470196563))"

## **Root Cause:**

When RabbitMQ reaches the vm\_memory\_high\_watermark threshold, it sets a memory resource limit alarm on the affected node and stops accepting new connections. The data-plane-agent service detects its connections to RabbitMQ are refused and marks itself as unhealthy, which causes the frontend error above.

To confirm this in the logs, retrieve the RabbitMQ container logs using one of the following methods.

**Recent versions of Domino** — stream logs directly from the Kubernetes container:

```
kubectl logs -n <your-domino-platform-namespace> rabbitmq-ha-311-0 --follow
```

> **Note:** The pod name rabbitmq-ha-311-0 is illustrative. Replace it with the name of the affected pod in your environment.

**Older versions of Domino** — exec into the container and read the log file:

```
kubectl exec -it -n <your-domino-platform-namespace> rabbitmq-ha-311-0 -c rabbitmq-ha -- /bin/bash
cat /var/log/rabbitmq/*log
```

Look for vm\_memory\_high\_watermark set and memory resource limit alarm set entries to confirm the root cause.

## **Resolution:**

**Temporary fix**

Restart the affected RabbitMQ pod to clear the alarm and restore operations. Use the following command, replacing the pod name with the affected pod in your environment:

```
kubectl delete pod -n <your-domino-platform-namespace> rabbitmq-ha-311-0
```

The pod restarts automatically. However, memory usage typically increases over time and the alarm recurs without a permanent configuration change.

**Permanent fix**

Increase the vm\_memory\_high\_watermark setting in the rabbitmq-ha ConfigMap to set an absolute memory limit appropriate for your container's resource limits.

The default value is vm\_memory\_high\_watermark.relative = 0.4, which allocates 40% of available memory. This default protects against out-of-memory (OOM) situations. Where container memory limits allow, replacing the relative value with an explicit absolute value provides more predictable behavior. For example, on a container with a 6 GB memory limit, 3500MB is an acceptable threshold.

**Step 1: Edit the ConfigMap.**

```
kubectl edit cm -n <your-domino-platform-namespace> rabbitmq-ha
```

**Step 2: Replace the relative setting with an absolute value.**

```
## Memory Threshold
# vm_memory_high_watermark.relative = 0.4
vm_memory_high_watermark.absolute = 3500MB
```

**Step 3: Restart the RabbitMQ pods to apply the change.** Run the following command to perform a rolling restart of the StatefulSet:

```
kubectl rollout restart statefulset -n <your-domino-platform-namespace> rabbitmq-ha
```

**Step 4: Verify the new watermark is active.** After the pods restart, check the logs for confirmation that the updated value loaded:

```
kubectl logs -n <your-domino-platform-namespace> <rabbitmq-pod-name> | grep vm_memory_high_watermark
```

Confirm the output references the absolute value you set (for example, 3500MB) rather than the previous relative value.


## Related topics

- [Troubleshooting](/knowledge-base/troubleshooting-overview.md)
- [Control plane security](/cloud/admin/security-and-compliance/control-plane-security.md)
- [Asynchronous Domino endpoints capacity planning](/cloud/admin/architecture-and-deployment/sizing-the-infrastructure/size-async-domino-endpoint.md)
- [Nexus hybrid architecture](/cloud/admin/architecture-and-deployment/architecture/nexus-hybrid-architecture.md)