Support Questions
Find answers, ask questions, and share your expertise

Does Spark --proxy-user not work for client deploy mode on CDH 6.3?

Highlighted

Does Spark --proxy-user not work for client deploy mode on CDH 6.3?

Explorer

Greetings,

 

I'm trying to diagnose an issue I'm seeing that is specific to CDH 6.3.  This is a two node Kerberized cluster.  I am attempting to submit a Spark application, using --proxy-user, and finding that this only works with cluster deploy mode, not client, which is odd.  From a client node on the cluster (called node-1.cluster), I am running the following shell session:

 

 

# first, kinit as a valid principal; this is required for --proxy-user to work at all
kinit -kt /path/to/my.keytab princ@CLUSTER
# now, run the SparkPi exampe, with a proxy-user specified as "bob", in client mode
# bob is also configured in the CDH settings under hadoop.proxyuser.princ.users

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode client \
  --executor-memory 1G \
  --proxy-user bob \
  --num-executors 1 \
  /opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_2.11-2.4.0-cdh6.3.0.jar \
  1000

# this runs for a bit, but the fails with...

19/12/13 16:18:11 ERROR cluster.YarnClientSchedulerBackend: Diagnostics message: Uncaught exception: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client c
annot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "node-2.cluster/172.18.0.3"; destination host is: "node-1.cluster":8020;
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:808)
        at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1503)
        at org.apache.hadoop.ipc.Client.call(Client.java:1445)
        at org.apache.hadoop.ipc.Client.call(Client.java:1355)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
<snip>
Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
<snip>
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
        at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
        at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
        at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)
        at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
        at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
        ... 38 more

# now, run the exact same command using cluster deploy mode instead; this succeeds

spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master yarn \
  --deploy-mode cluster \
  --executor-memory 1G \
  --proxy-user bob \
  --num-executors 1 \
  /opt/cloudera/parcels/CDH/lib/spark/examples/jars/spark-examples_2.11-2.4.0-cdh6.3.0.jar \
  1000

 

 

Now, repeating the exact same procedure outlined above on CDH 6.1, both deploy modes succeed.  Any ideas why this might be the case?

 

2 REPLIES 2
Highlighted

Re: Does Spark --proxy-user not work for client deploy mode on CDH 6.3?

New Contributor

Hi Jeff

 

Did you manage to get this client mode working with --proxy-user?

I am still seeing this issue on cdh6.2.1

It seems to work fine with cluster mode and with opensource Spark.

Re: Does Spark --proxy-user not work for client deploy mode on CDH 6.3?

Community Manager

Hi @BlueBerret as this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post.


Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community: