Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How can I enable Kerberos debug logging for Spark?

avatar

I tried the following command but in the yarn application log, no extra kerberos debug logging.

spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi /usr/hdp/current/spark-client/lib/spark-examples-*.jar 1 --conf spark.driver.extraJavaOptions = "-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true"

1 ACCEPTED SOLUTION

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar
Super Guru
@Hajime

did you try setting debug args for executors spark.driver.extraJavaOptions?

avatar

Do you mean spark.executor.extraJavaOptions?

If so, yes i tried.

avatar
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
New Contributor

Hi,

What changes you made to log4j.properties file to enable kerberos debug mode?

avatar
Master Guru

@RajaChintala You have to make below changes.

HADOOP_OPTS="-Dsun.security.krb5.debug=true"

For more details follow below docs:

  1. https://docs.cloudera.com/documentation/enterprise/5-10-x/topics/cdh_sg_debug_sun_kerberos_enable.ht...
  2. https://spark.apache.org/docs/2.0.0/running-on-yarn.html#troubleshooting-kerberos
  3. https://spark.apache.org/docs/2.0.0/running-on-yarn.html#troubleshooting-kerberos

Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.