Support Questions
Find answers, ask questions, and share your expertise

Verbose Kerberos authentication logs for spark-submit

Verbose Kerberos authentication logs for spark-submit

New Contributor

When submitting a Spark job to a cluster which is configured with Kerberos authentication, we pass `--principal` and `--keytab` parameters to `spark-submit` which it uses to authenticate and generate a ticket for secure access to the Hadoop cluster.


In order to debug issues with authentication, usually one would pass the JVM parameter `` when executing a `java` task to get verbose logs from the Kerberos libraries. How can such an option be passed to `spark-submit` itself, so that we can debug any issues with authentication?


N.B. It is not sufficient to pass the flag to `spark.driver.extraJavaOptions` or `spark.executor.extraJavaOptions` because when running with `--deploy-mode cluster` the only code which runs on the client machine is the `spark-submit` code (the driver and executor, which would run on the cluster, wouldn't even be started in the case of a failure to authenticate to run on the secure cluster!).