question Re: How can I enable Kerberos debug logging for Spark? in Archives of Support Questions (Read Only)

How can I enable Kerberos debug logging for Spark?

hosako — Tue, 24 Jan 2017 12:39:23 GMT

I tried the following command but in the yarn application log, no extra kerberos debug logging.

spark-submit --master yarn-cluster --class org.apache.spark.examples.SparkPi /usr/hdp/current/spark-client/lib/spark-examples-*.jar 1 --conf spark.driver.extraJavaOptions = "-Dsun.security.krb5.debug=true -Dsun.security.spnego.debug=true"

Re: How can I enable Kerberos debug logging for Spark?

rajkumar_singh — Tue, 24 Jan 2017 13:41:04 GMT

@Hajime

did you try setting debug args for executors spark.driver.extraJavaOptions?

Re: How can I enable Kerberos debug logging for Spark?

hosako — Tue, 24 Jan 2017 13:43:01 GMT

Do you mean spark.executor.extraJavaOptions?

If so, yes i tried.

Re: How can I enable Kerberos debug logging for Spark?

hosako — Tue, 24 Jan 2017 14:58:06 GMT

It seems I needed to change /etc/spark/conf/log4j.properties to enable DEBUG ...

Re: How can I enable Kerberos debug logging for Spark?

RajaChintala — Fri, 11 Sep 2020 17:00:01 GMT

Hi,

What changes you made to log4j.properties file to enable kerberos debug mode?

Re: How can I enable Kerberos debug logging for Spark?

GangWar — Mon, 14 Sep 2020 07:16:20 GMT

@RajaChintala You have to make below changes.

HADOOP_OPTS="-Dsun.security.krb5.debug=true"

For more details follow below docs: