Architecture
In addition to the Prometheus server running in a non-Nexus Domino cluster, remote data plane monitoring will deploy and run a pod in each data plane at install time running OpenTelemetry Collector, also called Otel Collector. In a remote data plane, the OpenTelemetry Collector pod is configured to scrape metrics from Domino components, Kubernetes, and the underlying infrastructure and then forward the collected metrics to the OpenTelemetry Collector running in the control plane. Metrics sent from each remote data plane have adata_plane_id label containing the ID of the data plane they were collected from.
The OpenTelemetry Collector running in the control plane (or local data plane) is responsible for receiving connections from OpenTelemetry Collectors running in each of the remote data planes and forwarding it to the Prometheus server. Once ingested into Prometheus, the remote data plane metrics are available for use in alerting and monitoring just like any other metric collected from the local data plane.
Communication between OpenTelemetry Collectors is secured via TLS and basic Auth authentication. The secret used for the authentication is stored in the secret named domino-otel-collector-http-auth.