Skip to main content
The fleetcommand-agent image that runs operator jobs creates definitions for an additional custom resource definition managed by the platform operator called HelmRelease. These resources map one-to-one to the Helm releases that are deployed to the cluster and are managed by Domino. They abide by a separate reconciliation loop than the Domino resource and are continuously evaluated for drift between the deployed manifest of the Helm release and the live state of the cluster. While the operator is capable of correcting drift, this behavior is not yet enabled globally or configurable by service through the Domino resource. By default, it will warn of drift on the HelmRelease resource conditions directly. In a future release, this will be surfaced as a configurable option. Using ddlctl is the best way to inspect the state of HelmRelease resources in your cluster: 
# Get all HelmRelease resources in the cluster across namespaces
ddlctl get helmrelease --all

# Get all HelmRelease resources in the domino-platform namespace
ddlctl get helmrelease --namespace domino-platform

# Get all HelmRelease resources in the cluster that are marked as Stalled
ddlctl get helmrelease --all --status stalled=true
A HelmRelease is marked as Stalled when the operator detects that:
  • the Helm release has drifted from the desired state,
  • the Helm release is in a Failed state,
  • the Helm release is locked in a pending state,
  • the Helm release was deleted, or
  • the latest Helm revision does not match the desired revision of the current Domino generation.
To get all HelmRelease resources in the cluster that are marked as Ready, run the following: 
ddlctl get helmrelease --all --status ready=true
HelmRelease resources are deployed with a default 5 minute interval, meaning if a release were to get out of sync in the cluster it will not necessarily register as drift immediately, but get picked up on the next reconciliation. If you want to force a reconciliation, you can do this through the ddlctl command line: 
ddlctl reconcile helmrelease nucleus -n domino-platform

​
Investigating drift

Discovering what has drifted on a HelmRelease resource can be done in a few ways. ddlctl offers a subcommand for inspecting the diff of a Helm release against the live state of the cluster: 
ddlctl diff helmrelease nucleus -n domino-platform
If the resource has drifted, you can expect to see something similar to the following: 
NAME                    READY   REASON  MESSAGE                                             DRIFT DETECTION MODE    SUSPENDED
domino-data-importer    False   Drifted cluster has drifted from desired helmrelease state  warn                    false
For resources that are in sync, you can expect to see something more like the following: 
NAME    READY   REASON  MESSAGE                                     DRIFT DETECTION MODE    SUSPENDED
nucleus True    InSync  helmrelease is in sync with cluster state   warn                    false
The operator also writes information on the nature of drift to events, which can be inspected with kubectl describe, i.e.: 
# Inspect the events of a Helm release
kubectl describe helmrelease nucleus -n domino-platform
The Warning event will report on the resource where drift was detected, the type of drift, and include the JSON patch (either in full or in part) that would be applied if correct mode were enabled on the HelmRelease resource rather than warn.
As there is a character limit on Kubernetes events, the JSON patch will be truncated to 500 characters max, but the full patch can be found in the operator logs, which can also be accessed with ddlctl by running ddlctl logs operator.