question Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure in Support Questions

After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

raju_ramakrishn — Wed, 07 Dec 2016 18:05:04 GMT

HBase is throwing an exception after enabling Kerberos-

2016-12-07 10:33:07,963 ERROR [main-SendThread(y.server.com:2181)] client.ZooKeeperSaslClient: SASL authentication failed using login context 'Client'.
2016-12-07 10:33:08,068 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2290)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2304)
Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hbase-secure
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:576)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:555)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1313)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1291)

I connected to zookepeer with the following command, and couldn't find the "hbase-secure" directory created. Only "hbase" directory exists -

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server x.server.com,y.server.com,z.server.com get /

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

elserj — Thu, 08 Dec 2016 00:23:30 GMT

Check the HBase master log for additional information about the ZooKeeper Kerberos login. You should see information shortly after the process starts which prints the ticket lifetime information. There may be other exceptions in the log about failure to login to Kerberos that result in this znode creation failing.

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

raju_ramakrishn — Thu, 08 Dec 2016 04:16:24 GMT

Thanks @Josh Elser

I analyzed the issue further, and found that the problem in Zookeeper SASL. After kerberos, Zookeeper is expecting the port number 2888-3888 to be opened between all the 3 Zookeper servers. However, I hadn't opened that range of ports. Hence SASL error was thrown even with a simple ./zkCli.sh command. I have asked the customer to open the port range.

Please let me know if this is not correct.

Regards,

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

elserj — Thu, 08 Dec 2016 04:48:45 GMT

Your terminology is off, but the explanation seems plausible :). 2888-3888 is the range used by ZK internal communication (ZK servers talking to each other). I can imagine that if ZK servers couldn't communicate with each other, ZK would not operate as expected. SASL is just way of performing authentication and has nothing to do with the low-level transport over the wire.

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

raju_ramakrishn — Thu, 08 Dec 2016 07:33:18 GMT

Thank you @Josh Elser 🙂