After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

[#1]

HBase is throwing an exception after enabling Kerberos-

2016-12-07 10:33:07,963 ERROR [main-SendThread(y.server.com:2181)] client.ZooKeeperSaslClient: SASL authentication failed using login context 'Client'.

2016-12-07 10:33:08,068 ERROR [main] master.HMasterCommandLine: Master exiting

java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster

at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2290)

at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233)

at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2304)

Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hbase-secure

at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)

at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:576)

at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:555)

at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1313)

at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1291)

I connected to zookepeer with the following command, and couldn't find the "hbase-secure" directory created. Only "hbase" directory exists -

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server x.server.com,y.server.com,z.server.com get /

[#2 · Accepted solution]

Check the HBase master log for additional information about the ZooKeeper Kerberos login. You should see information shortly after the process starts which prints the ticket lifetime information. There may be other exceptions in the log about failure to login to Kerberos that result in this znode creation failing.

[#3]

Thanks @Josh Elser

I analyzed the issue further, and found that the problem in Zookeeper SASL. After kerberos, Zookeeper is expecting the port number 2888-3888 to be opened between all the 3 Zookeper servers. However, I hadn't opened that range of ports. Hence SASL error was thrown even with a simple ./zkCli.sh command. I have asked the customer to open the port range.

Please let me know if this is not correct.

Regards,

[#4]

Your terminology is off, but the explanation seems plausible :). 2888-3888 is the range used by ZK internal communication (ZK servers talking to each other). I can imagine that if ZK servers couldn't communicate with each other, ZK would not operate as expected. SASL is just way of performing authentication and has nothing to do with the low-level transport over the wire.

[#5]

Thank you @Josh Elser 🙂