I have been trying to use a run a spark-submit command using spark-cdh 6.2.1 with proxy user.
(Note: First kinit using a principal name and keytab, then run the below command using a proxy user abc)
../CDH/bin/spark-submit --class org.apache.spark.example.SparkPi --master yarn --deploy-mode client --proxy-user abc /CDH/cloudera/parcels/CDH/lib/spark/examples/jar/spark-example*.jar 10
The above command works successfully in cluster mode.
However in client mode , it goes to accepted state and then finally errors with:
AM container is launched, waiting for AM container to Register with RM
ipc.Client: Exception encountered while connecting to the server: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEn,KERBEROS]
local host is: "abc.server.org/<ip address>", destination host is: "bcd.server.org/<ip address>"
I have also tested , the same command with open source version of Spark 2.4.6, the same Spark Submit command works in both client and cluster mode. (Everything else being exactly the same)
What am I hitting this issue when trying to use CDH version of Spark alone? Is there any parameter that can set to have a workaround this issue?
@BlueBerret This seems a delegation token issue. Are you able to kinit successfully? Also check the DNS resolution between hosts given in below error.