Does Spark --proxy-user not work for client deploy mode on CDH 6.3?

[#1]

Greetings,

I'm trying to diagnose an issue I'm seeing that is specific to CDH 6.3.  This is a two node Kerberized cluster.  I am attempting to submit a Spark application, using --proxy-user, and finding that this only works with cluster deploy mode, not client, which is odd.  From a client node on the cluster (called node-1.cluster), I am running the following shell session:

# first, kinit as a valid principal; this is required for --proxy-user to work at all
kinit -kt /path/to/my.keytab princ@CLUSTER
# now, run the SparkPi exampe, with a proxy-user specified as "bob", in client mode
# bob is also configured in the CDH settings under hadoop.proxyuser.princ.users

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode client \
  --executor-memory 1G \
  --proxy-user bob \
  --num-executors 1 \
  /opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_2.11-2.4.0-cdh6.3.0.jar \
  1000

# this runs for a bit, but the fails with...

19/12/13 16:18:11 ERROR cluster.YarnClientSchedulerBackend: Diagnostics message: Uncaught exception: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client c
annot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "node-2.cluster/172.18.0.3"; destination host is: "node-1.cluster":8020;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
        at org.apache.hadoop.ipc.Client.call(Client.java:1445)
        at org.apache.hadoop.ipc.Client.call(Client.java:1355)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
<snip>
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
<snip>
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)
        at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
        ... 38 more

# now, run the exact same command using cluster deploy mode instead; this succeeds

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 1G \
  --proxy-user bob \
  --num-executors 1 \
  /opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_2.11-2.4.0-cdh6.3.0.jar \
  1000

Now, repeating the exact same procedure outlined above on CDH 6.1, both deploy modes succeed.  Any ideas why this might be the case?

[#2]

Hi Jeff

Did you manage to get this client mode working with --proxy-user?

I am still seeing this issue on cdh6.2.1

It seems to work fine with cluster mode and with opensource Spark.

[#3]

Hi @BlueBerret as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.