> ## Documentation Index > Fetch the complete documentation index at: https://docs.domino.ai/llms.txt > Use this file to discover all available pages before exploring further. # Spark Executor Pods Unable to Find Kerberos Credentials Cache > Kerberos klist: No credentials cache found on Spark executors. Fix: set KRB5CCNAME and KRB5_CONFIG via ENV in the Dockerfile, run kinit in Pre-Run, and concatenate all extraJavaOptions in one line. ### ***Version/Environment (if relevant):*** Applies to all versions of Domino. ## **Issue:** When using Kerberos authentication with PySpark in a Domino Spark cluster to connect to a SQL Server, the following errors are encountered: ``` Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 1) (12.34.567.89 executor 0): com.microsoft.sqlserver.jdbc.SQLServerException: Integrated authentication failed. ClientConnectionId:1234d56a-78c9-1234-beda-eedc1234b567 ``` The [Spark configuration](https://docs.dominodatalab.com/en/latest/user_guide/f11f6a/work-with-your-cluster/#_spark_executors_versus_workers) has been set at both the code level and on the Spark Web UI's Project Integration page. Running `kinit` in the Spark driver (workspace) succeeds, and the token is visible via `klist` in the workspace logs: ``` I have no name!@spark-123456cf7c89123b456fa789-spark-worker-2:/tmp$ klist Ticket cache: FILE:/tmp/krb5cc_12345 Default principal: sys-data@AD.COMPANY.COM Valid starting Expires Service principal 03/29/23 15:28:44 03/29/23 23:28:44 krbtgt/AD.COMPANY.COM@AD.COMPANY.COM renew until 04/03/23 15:28:44 ``` However, `klist` fails on the executor Spark worker pods with the following error: ``` I have no name!@spark-1234567f1c23456b789faed1-spark-worker-0:/opt/bitnami/spark$ klist klist: No credentials cache found (filename: /tmp/krb5cc_12345) ``` ## **Root Cause:** There are two root causes: 1. Setting environment variables via `spark.executorEnv.ENVNAME` does not work. Environment variables must be set using `ENV` in the cluster environment's Dockerfile. 2. When setting multiple `extraJavaOptions` parameters with one line per option, only the last parameter is applied at runtime — each new line overwrites the previous one. ## **Resolution:** **1. Dockerfile Instructions** Set the following environment variables in the executor cluster environment's Dockerfile: * `ENV KRB5CCNAME=FILE:/mnt/data/workspace_name/krb5cc_12345` * `ENV KRB5_CONFIG=/mnt/data/workspace_name/krb5.conf` This changes the default `ccache` directory to a shared directory accessible by both the driver and executors, allowing executors to read the `ccache` generated by the driver during pre-run. Setting `KRB5CCNAME` via `spark.executorEnv.KRB5CCNAME` or any other method will not work — executors will always fall back to the default location `/tmp/krb5cc_12345`. Use `ENV` instead of `ARG` when setting `KRB5_CONFIG`. **2. Pre-Run Script** Run `kinit` in the Spark driver's environment Pre-Run script to generate and cache the Kerberos token on the dataset so that executors can use it for authentication. **3. `extraJavaOptions` Parameters** Under **Project Settings > Integrations > Apache Spark mode > Spark Configuration Options**, setting multiple `extraJavaOptions` with one line per parameter will not work, as each line overwrites the previous: ``` spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false") spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf") spark_conf.set("spark.executor.extraJavaOptions","-Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf") spark_conf.set("spark.executor.extraJavaOptions","-Dsun.security.krb5.debug=true") ![Project_settings_-_Integrations_-_Apache_Spark_mode.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr&ids=068UY00000Hru0q&d=%2Fa%2FUY0000018p1k%2FT0zoitoAOVJ0hrugfPbV4jh50.msvIGPLAufFbN3Yto&asPdf=false) ``` Instead, concatenate all `spark.driver.extraJavaOptions` parameters into a single line, separated by spaces: ``` spark_conf.set("spark.executor.extraJavaOptions","-Djavax.security.auth.useSubjectCredsOnly=false -Djava.security.auth.login.config=/mnt/data/workspace_name/SQLJDBCDriver2.conf -Djava.security.krb5.conf=/mnt/data/workspace_name/krb5.conf -Dsun.security.krb5.debug=true") ``` ![Spark\_Web\_UI\_-Environment-Spark\_Properties-\_all\_params.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000Hru0s\&d=%2Fa%2FUY0000018p1m%2FT8Et6o60_ob2jCkG.RVRJ9ufONxIQFEYytdAYaz6AMM\&asPdf=false) You can verify all parameters are correctly applied by checking **Spark Web UI > Environment > Spark Properties**. ![Spark\_Web\_UI\_-Env-Spark\_Properties-\_only\_last\_param.png](https://dominodatalab.file.force.com/sfc/dist/version/download/?oid=00D15000000EQXr\&ids=068UY00000Hru0r\&d=%2Fa%2FUY0000018p1l%2FrjC1BtmO6uZkRsyiSCYp8yG8pzZs6_0.3q_rqBYq8FQ\&asPdf=false) ### **Notes/Information:** Additional resources: * [Apache Spark 3.0 — Kerberos ticket cache](https://spark.apache.org/docs/3.0.0-preview/security.html#using-a-ticket-cache): By setting `spark.kerberos.renewal.credentials` to `ccache`, the local Kerberos ticket cache will be used for authentication. The location can be customized via the `KRB5CCNAME` environment variable. * [Spark on Domino — Configure Prerequisites](https://docs.dominodatalab.com/en/latest/user_guide/1962f3/configure-prerequisites/): Configure PySpark compute environments for workspaces and/or jobs connecting to your cluster. * [Domino Kerberos Authentication](https://docs.dominodatalab.com/en/latest/user_guide/ed5f5a/kerberos-authentication/): Domino supports keytab-based Kerberos authentication at the project or user level. Once configured, runs will automatically execute `kinit` and retrieve the ticket for authentication. ### **References/Internal Records:** ## Related topics - [Troubleshooting](/knowledge-base/troubleshooting-overview.md) - [Hadoop and Spark](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/index.md) - [Kerberos authentication](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/hadoop-and-spark/kerberos-authentication.md) - [Access data with Spark](/cloud/platform-capabilities/features/compute-clusters-hpc/spark/on-demand-spark/work-with-data.md)