Skip to main content
The fleetcommand-agent image that runs operator jobs creates definitions for an additional custom resource definition managed by the platform operator called HelmRelease. These resources map one-to-one to the Helm releases that are deployed to the cluster and are managed by Domino. They abide by a separate reconciliation loop than the Domino resource and are continuously evaluated for drift between the deployed manifest of the Helm release and the live state of the cluster. While the operator is capable of correcting drift, this behavior is not yet enabled globally or configurable by service through the Domino resource. By default, it warns of drift on the HelmRelease resource conditions directly. Domino plans to surface this as a configurable option in a future release. Using ddlctl is the best way to inspect the state of HelmRelease resources in your cluster:
A HelmRelease is marked as Stalled when the operator detects that:
  • the Helm release has drifted from the desired state,
  • the Helm release is in a Failed state,
  • the Helm release is locked in a pending state,
  • the Helm release was deleted, or
  • the latest Helm revision does not match the desired revision of the current Domino generation.
To get all HelmRelease resources in the cluster that are marked as Ready, run the following:
HelmRelease resources are deployed with a default 5 minute interval, meaning if a release were to get out of sync in the cluster it does not necessarily register as drift immediately, but gets picked up on the next reconciliation. If you want to force a reconciliation, you can do this through the ddlctl command line:

​
Investigating drift

Discovering what has drifted on a HelmRelease resource can be done in a few ways. ddlctl offers a subcommand for inspecting the diff of a Helm release against the live state of the cluster:
If the resource has drifted, you can expect to see something similar to the following:
For resources that are in sync, you can expect to see something more like the following:
The operator also writes information on the nature of drift to events, which can be inspected with kubectl describe, i.e.:
The Warning event reports on the resource where drift was detected, the type of drift, and includes the JSON patch (either in full or in part) that would be applied if correct mode were enabled on the HelmRelease resource rather than warn.
As there is a character limit on Kubernetes events, the JSON patch is truncated to 500 characters max, but the full patch can be found in the operator logs, which can also be accessed with ddlctl by running ddlctl logs operator.
Last modified on July 16, 2026

Related topics

Domino endpoint drift detectionDrift detection for monitored modelsTroubleshoot the Model MonitorAnalyze model quality