Unable to spark-submit using spark cdh 6.2.1 in yarn client mode with --proxy-user

[#1]

Hi

I have been trying to use a run a spark-submit command using spark-cdh 6.2.1 with proxy user.

(Note: First kinit using a principal name and keytab, then run the below command using a proxy user abc)

../CDH/bin/spark-submit --class org.apache.spark.example.SparkPi --master yarn --deploy-mode client   --proxy-user abc /CDH/cloudera/parcels/CDH/lib/spark/examples/jar/spark-example*.jar 10

The above command works successfully in cluster mode.

However in client mode , it goes to accepted state and then finally errors with:

Yarn Diagnostics:

AM container is launched, waiting for AM container to Register with RM

ipc.Client: Exception encountered while connecting to the server: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEn,KERBEROS]

local host is: "abc.server.org/<ip address>", destination host is: "bcd.server.org/<ip address>"

I have also tested , the same command  with open source version of Spark 2.4.6, the same Spark Submit command works in both client and cluster mode. (Everything else being exactly the same)

What am I hitting this issue when trying to use CDH version of Spark alone? Is there any parameter that can set to have a workaround this issue?

Thanks

[#2]

@BlueBerret This seems a delegation token issue. Are you able to kinit successfully? Also check the DNS resolution between hosts given in below error. 

local host is: "abc.server.org/<ip address>", destination host is: "bcd.server.org/<ip address>"