> ## Documentation Index
> Fetch the complete documentation index at: https://docs.domino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Drift Detection

The `fleetcommand-agent` image that runs operator jobs creates definitions for an additional custom resource definition managed by the platform operator called `HelmRelease`. These resources map one-to-one to the Helm releases that are deployed to the cluster and are managed by Domino. They abide by a separate reconciliation loop than the `Domino` resource and are continuously evaluated for drift between the deployed manifest of the Helm release and the live state of the cluster.

While the operator is capable of correcting drift, this behavior is not yet enabled globally or configurable by service through the `Domino` resource. By default, it will warn of drift on the `HelmRelease` resource conditions directly. In a future release, this will be surfaced as a configurable option.

Using `ddlctl` is the best way to inspect the state of `HelmRelease` resources in your cluster:

```shell theme={null}
# Get all HelmRelease resources in the cluster across namespaces
ddlctl get helmrelease --all

# Get all HelmRelease resources in the domino-platform namespace
ddlctl get helmrelease --namespace domino-platform

# Get all HelmRelease resources in the cluster that are marked as Stalled
ddlctl get helmrelease --all --status stalled=true
```

A `HelmRelease` is marked as `Stalled` when the operator detects that:

* the Helm release has drifted from the desired state,

* the Helm release is in a `Failed` state,

* the Helm release is locked in a pending state,

* the Helm release was deleted, or

* the latest Helm revision does not match the desired revision of the current `Domino` generation.

To get all HelmRelease resources in the cluster that are marked as `Ready`, run the following:

```shell theme={null}
ddlctl get helmrelease --all --status ready=true
```

`HelmRelease` resources are deployed with a default `5` minute interval, meaning if a release were to get out of sync in the cluster it will not necessarily register as drift immediately, but get picked up on the next reconciliation.

If you want to **force** a reconciliation, you can do this through the `ddlctl` command line:

```shell theme={null}
ddlctl reconcile helmrelease nucleus -n domino-platform
```

## Investigating drift

Discovering what has drifted on a `HelmRelease` resource can be done in a few ways.

`ddlctl` offers a subcommand for inspecting the diff of a Helm release against the live state of the cluster:

```shell theme={null}
ddlctl diff helmrelease nucleus -n domino-platform
```

If the resource has drifted, you can expect to see something similar to the following:

```
NAME                    READY   REASON  MESSAGE                                             DRIFT DETECTION MODE    SUSPENDED
domino-data-importer    False   Drifted cluster has drifted from desired helmrelease state  warn                    false
```

For resources that are in sync, you can expect to see something more like the following:

```
NAME    READY   REASON  MESSAGE                                     DRIFT DETECTION MODE    SUSPENDED
nucleus True    InSync  helmrelease is in sync with cluster state   warn                    false
```

The operator also writes information on the nature of drift to events, which can be inspected with `kubectl describe`, i.e.:

```shell theme={null}
# Inspect the events of a Helm release
kubectl describe helmrelease nucleus -n domino-platform
```

The `Warning` event will report on the resource where drift was detected, the type of drift, and include the JSON patch (either in full or in part) that would be applied if `correct` mode were enabled on the `HelmRelease` resource rather than `warn`.

<Note>
  As there is a character limit on Kubernetes events, the JSON patch will be truncated to 500 characters max, but the full patch can be found in the operator logs, which can also be accessed with `ddlctl` by running `ddlctl logs operator`.
</Note>