We are using HiveContext in a Spark Application and running it on a secure cluster with Sentry enabled. We are on CDH 5.8
In our use case we are using proxy-user for impersonation.
spark-submit \
--class fire.execute.WorkflowExecuteFromFile \
--keytab /disk01/sparkflows/release/fire-ui/sparkflows.keytab \
--proxy-user cloudera --master yarn \
--deploy-mode client \
/disk01/sparkflows/release/fire/core/target/fire-core-1.3.0-jar-with-dependencies.jar
We are running into the exception below. Though SPARK-13478 is in CDH 5.7.
http://archive.cloudera.com/cdh5/cdh/5/spark-1.6.0-cdh5.7.0.releasenotes.html
https://issues.apache.org/jira/browse/SPARK-13478
org.apache.hadoop.hive.shims.Hadoop23Shims for Hadoop version 2.6.0-cdh5.7.0 16/10/15 12:15:56 INFO hive.metastore: Trying to connect to metastore with URI thrift://venice.hadoop:9083 16/10/15 12:15:56 ERROR transport.TSaslTransport: SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at
To ensure everything else is good on the cluster, we tested SparkPi. It runs successfully without any issues:
spark-submit \
--class org.apache.spark.examples.SparkPi \
--keytab /disk01/sparkflows/release/fire-ui/sparkflows.keytab \
--proxy-user cloudera --master yarn \
--deploy-mode client \
/opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/jars/spark-examples-1.6.0-cdh5.7.0-hadoop2.6.0-cdh5.7.0.jar 10
Thanks,
Jayant
