Invalid Client Credentials When Generating A Service Account Token

Overview

When creating a service account in Domino, you may encounter the following error while generating a token:

{"message":"Request unauthorized: {\"error\":\"unauthorized_client\",\"error_description\":\"Invalid client or Invalid client credentials\"}","success":false}

This typically indicates a mismatch between the client secret stored in Kubernetes and the one configured in Keycloak for the domino-service-accounts client.

Symptoms

Service account creation succeeds, but token generation fails.
Error response shows unauthorized_client or Invalid client credentials.
Nucleus frontend logs display OidcUnauthorizedException related to the domino-service-accounts client.

2024-11-12T09:32:09.048739313-05:00 domino.helpers.oidc.OidcUnauthorizedException: Request unauthorized: {"error":"unauthorized_client","error_description":"Invalid client or Invalid client credentials"}

Issue may appear after making changes to Keycloak configuration (e.g., adding or modifying JWT keys).

Root Cause

The domino-service-accounts client credentials in Keycloak and the Kubernetes secret may become out of sync. Since Keycloak client secrets cannot be directly edited, and Kubernetes secrets are auto-generated, reconciliation is required to realign both values.

How to Verify Credentials

Fetch values from the cluster:

# Get clientId
kubectl -n domino-platform get secret nucleus-domino-service-accounts-client -o jsonpath='{.data.clientId}' | base64 -d; echo

# Get clientSecret
kubectl -n domino-platform get secret nucleus-domino-service-accounts-client -o jsonpath='{.data.clientSecret}' | base64 -d; echo

The clientId should be domino-service-accounts.
Note down the clientSecret.

2. Check Keycloak configuration:

Log in to Keycloak.
Go to: Domino Realm > Clients > domino-service-accounts > Credentials.
Reveal the Client Secret and compare it with the one from Kubernetes.

! If they do not match, reconciliation is needed.

Reconciliation Procedure

Since neither Keycloak nor Kubernetes allows direct editing of these secrets, the recommended approach is to re-import the domino-service-accounts client into Keycloak: 1. Export the client configuration:

In Keycloak UI, go to: Domino Realm > Clients > domino-service-accounts > Action > Export.

Save the JSON file.

2. Update the exported file:

Locate the "secret" property in the JSON.
Replace its value with the clientSecret obtained from Kubernetes.

"secret": "your-cluster-secret-value"

! 3. Delete the old client:

In Keycloak UI, navigate to Domino Realm > Clients > domino-service-accounts > Action > Delete.

4. Re-import the updated client:

Go to Domino Realm > Clients > Clients List > Import Client.
Upload the modified JSON file.

Click Save

5. Verify the reconciliation:

Navigate back to Credentials for the domino-service-accounts client.
Confirm that the secret matches the value in Kubernetes.
Retry generating the service account token.

Important Notes

Avoid toggling the Client Authentication setting in Keycloak unnecessarily. Doing so may regenerate the secret and cause mismatches.
If new JWT keys are required (e.g., for S3 connectivity), ensure existing Domino-related client configurations are not modified.
After reconciliation, service account token generation should work again without impacting other Keycloak configurations.

Domino Service Accounts Guide

​Overview

​Symptoms

​Root Cause

​How to Verify Credentials

​Reconciliation Procedure

​Important Notes

​Related Documentation