How can I enable Kerberos debug logging for Spark?

[#1]

I tried the following command but in the yarn application log, no extra kerberos debug logging.

spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi /usr/hdp/current/spark-client/lib/spark-examples-*.jar 1 --conf spark.driver.extraJavaOptions = "-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true"

[#2]

@Hajime

did you try setting debug args for executors spark.driver.extraJavaOptions?

[#3]

Do you mean spark.executor.extraJavaOptions?

If so, yes i tried.

[#4 · Accepted solution]

It seems I needed to change /etc/spark/conf/log4j.properties to enable DEBUG ...

[#5]

Hi,

What changes you made to log4j.properties file to enable kerberos debug mode?

[#6]

@RajaChintala You have to make below changes.

HADOOP_OPTS="-Dsun.security.krb5.debug=true"

For more details follow below docs:

1. https://docs.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_sg_debug_sun_kerberos_enable.ht...
2. https://spark.apache.org/docs/2.0.0/running-on-yarn.html#troubleshooting-kerberos
3. https://spark.apache.org/docs/2.0.0/running-on-yarn.html#troubleshooting-kerberos