Spark Executor Pods Unable to Find Kerberos Credentials Cache

Version/Environment (if relevant):

Applies to all versions of Domino.

Issue:

When using Kerberos authentication with PySpark in a Domino Spark cluster to connect to a SQL Server, the following errors are encountered:

Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) (12.34.567.89 executor 0): com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:1234d56a-78c9-1234-beda-eedc1234b567

The Spark configuration has been set at both the code level and on the Spark Web UI’s Project Integration page. Running kinit in the Spark driver (workspace) succeeds, and the token is visible via klist in the workspace logs:

I have no name!@spark-123456cf7c89123b456fa789-spark-worker-2:/tmp$ klist
Ticket cache: FILE:/tmp/krb5cc_12345
Default principal: sys-data@AD.COMPANY.COM

Valid starting   Expires       Service principal
03/29/23 15:28:44  03/29/23 23:28:44  krbtgt/AD.COMPANY.COM@AD.COMPANY.COM
   renew until 04/03/23 15:28:44

However, klist fails on the executor Spark worker pods with the following error:

I have no name!@spark-1234567f1c23456b789faed1-spark-worker-0:/opt/bitnami/spark$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_12345)

Root Cause:

There are two root causes:

Setting environment variables via spark.executorEnv.ENVNAME does not work. Environment variables must be set using ENV in the cluster environment’s Dockerfile.
When setting multiple extraJavaOptions parameters with one line per option, only the last parameter is applied at runtime — each new line overwrites the previous one.

Resolution:

1. Dockerfile Instructions Set the following environment variables in the executor cluster environment’s Dockerfile:

ENV KRB5CCNAME=FILE:/mnt/data/workspace_name/krb5cc_12345
ENV KRB5_CONFIG=/mnt/data/workspace_name/krb5.conf

This changes the default ccache directory to a shared directory accessible by both the driver and executors, allowing executors to read the ccache generated by the driver during pre-run. Setting KRB5CCNAME via spark.executorEnv.KRB5CCNAME or any other method will not work — executors will always fall back to the default location /tmp/krb5cc_12345. Use ENV instead of ARG when setting KRB5_CONFIG. 2. Pre-Run Script Run kinit in the Spark driver’s environment Pre-Run script to generate and cache the Kerberos token on the dataset so that executors can use it for authentication. 3. extraJavaOptions Parameters Under Project Settings > Integrations > Apache Spark mode > Spark Configuration Options, setting multiple extraJavaOptions with one line per parameter will not work, as each line overwrites the previous:

spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false")
spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf")
spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf")
spark_conf.set("spark.executor.extraJavaOptions","-Dsun.security.krb5.debug=true")
![Project_settings_-_Integrations_-_Apache_Spark_mode.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr&ids=068UY00000Hru0q&d=%2Fa%2FUY0000018p1k%2FT0zoitoAOVJ0hrugfPbV4jh50.msvIGPLAufFbN3Yto&asPdf=false)

Instead, concatenate all spark.driver.extraJavaOptions parameters into a single line, separated by spaces:

spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf -Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf -Dsun.security.krb5.debug=true")

Spark_Web_UI_-Environment-Spark_Properties-_all_params.png

You can verify all parameters are correctly applied by checking Spark Web UI > Environment > Spark Properties. Spark_Web_UI_-Env-Spark_Properties-_only_last_param.png

Spark_Web_UI_-Env-Spark_Properties-_only_last_param.png

Notes/Information:

Additional resources:

Apache Spark 3.0 — Kerberos ticket cache: By setting spark.kerberos.renewal.credentials to ccache, the local Kerberos ticket cache will be used for authentication. The location can be customized via the KRB5CCNAME environment variable.
Spark on Domino — Configure Prerequisites: Configure PySpark compute environments for workspaces and/or jobs connecting to your cluster.
Domino Kerberos Authentication: Domino supports keytab-based Kerberos authentication at the project or user level. Once configured, runs will automatically execute kinit and retrieve the ticket for authentication.

Overview

Spark Executor Pods Unable to Find Kerberos Credentials Cache

Version/Environment (if relevant):

Issue:

Root Cause:

Resolution:

Notes/Information:

References/Internal Records:

​Version/Environment (if relevant):

​Issue:

​Root Cause:

​Resolution:

​Notes/Information:

​References/Internal Records:

Version/Environment (if relevant):

Issue:

Root Cause:

Resolution:

Notes/Information:

References/Internal Records: