Hi,
I am running a spark application in a Kerberos based HDP
platform. This spark application connects to HBase, write and read data
perfectly well in a local mode on any node in the cluster. However, when
I run this application on the cluster by using "-master yarn and
--deploymode client (or cluster)" the Kerberos authentication fails. I
have tried all sorts of things by doing Kinit outside of the application
on each node, and doing Kerberos authentication inside the application
as well but none of it has worked so far. In the local mode, nothing seems to have any issue and everything works: when I do kinit outside and do not perform any authentication inside the application. However, in the cluster mode nothing works whether I authenticate inside the application of outside the application. Here is an extract of the
stack trace:
ERROR ipc.AbstractRpcClient: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
Below is the code that I used for authenticating inside the application:
Configuration conf = HBaseConfiguration.create(); conf.addResource(new Path(hbaseConfDir,"hbase-site.xml")); conf.addResource(new Path(hadoopConfDir,"core-site.xml")); conf.set("hbase.client.keyvalue.maxsize", "0");conf.set("hbase.rpc.controllerfactory.class","org.apache.hadoop.hbase.ipc.RpcControllerFactory");
<b> conf.set("hadoop.security.authentication", "kerberos");
conf.set("hbase.security.authentication", "kerberos");
UserGroupInformation.setConfiguration(conf);
String keyTab="/etc/security/keytabs/somekeytab";
UserGroupInformation ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("name@xyz.com", keyTab);
UserGroupInformation.setLoginUser(ugi); </b>
connection=ConnectionFactory.createConnection(conf);
logger.debug("HBase connected"); Adding or removing the bold lines in the above code didn't really have any effect other than the fact that when the bold lines are there kinit outside of the application is not needed.
Please let me know how can I solve this problem. It has been quite some time I am hitting my head on this issue.
