Support Questions
Find answers, ask questions, and share your expertise

How can I enable Kerberos debug logging for Spark?

Solved Go to solution

How can I enable Kerberos debug logging for Spark?

I tried the following command but in the yarn application log, no extra kerberos debug logging.

spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi /usr/hdp/current/spark-client/lib/spark-examples-*.jar 1 --conf spark.driver.extraJavaOptions = "-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true"

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How can I enable Kerberos debug logging for Spark?

It seems I needed to change /etc/spark/conf/log4j.properties to enable DEBUG ...

View solution in original post

5 REPLIES 5

Re: How can I enable Kerberos debug logging for Spark?

@Hajime

did you try setting debug args for executors spark.driver.extraJavaOptions?

Re: How can I enable Kerberos debug logging for Spark?

Do you mean spark.executor.extraJavaOptions?

If so, yes i tried.

Re: How can I enable Kerberos debug logging for Spark?

It seems I needed to change /etc/spark/conf/log4j.properties to enable DEBUG ...

View solution in original post

Re: How can I enable Kerberos debug logging for Spark?

New Contributor

Hi,

What changes you made to log4j.properties file to enable kerberos debug mode?

Re: How can I enable Kerberos debug logging for Spark?

Master Collaborator

@RajaChintala You have to make below changes.

HADOOP_OPTS="-Dsun.security.krb5.debug=true"

For more details follow below docs:

  1. https://docs.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_sg_debug_sun_kerberos_enable.ht...
  2. https://spark.apache.org/docs/2.0.0/running-on-yarn.html#troubleshooting-kerberos
  3. https://spark.apache.org/docs/2.0.0/running-on-yarn.html#troubleshooting-kerberos

Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.