Support Questions

Find answers, ask questions, and share your expertise
Welcome to the upgraded Community! Read this blog to see What’s New!

Unable to spark-submit using spark cdh 6.2.1 in yarn client mode with --proxy-user

New Contributor



I have been trying to use a run a spark-submit command using spark-cdh 6.2.1 with proxy user.

(Note: First kinit using a principal name and keytab, then run the below command using a proxy user abc)


../CDH/bin/spark-submit --class org.apache.spark.example.SparkPi --master yarn --deploy-mode client   --proxy-user abc /CDH/cloudera/parcels/CDH/lib/spark/examples/jar/spark-example*.jar 10


The above command works successfully in cluster mode.

However in client mode , it goes to accepted state and then finally errors with:


Yarn Diagnostics:

AM container is launched, waiting for AM container to Register with RM


ipc.Client: Exception encountered while connecting to the server: Client cannot authenticate via:[TOKEn,KERBEROS]


local host is: "<ip address>", destination host is: "<ip address>"


I have also tested , the same command  with open source version of Spark 2.4.6, the same Spark Submit command works in both client and cluster mode. (Everything else being exactly the same)


What am I hitting this issue when trying to use CDH version of Spark alone? Is there any parameter that can set to have a workaround this issue?





@BlueBerret This seems a delegation token issue. Are you able to kinit successfully? Also check the DNS resolution between hosts given in below error. 

local host is: "<ip address>", destination host is: "<ip address>"

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.