Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

Solved Go to solution
Highlighted

After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

HBase is throwing an exception after enabling Kerberos-

2016-12-07 10:33:07,963 ERROR [main-SendThread(y.server.com:2181)] client.ZooKeeperSaslClient: SASL authentication failed using login context 'Client'.

2016-12-07 10:33:08,068 ERROR [main] master.HMasterCommandLine: Master exiting

java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster

at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2290)

at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233)

at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)

at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2304)

Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hbase-secure

at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)

at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)

at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)

at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:576)

at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:555)

at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1313)

at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1291)

I connected to zookepeer with the following command, and couldn't find the "hbase-secure" directory created. Only "hbase" directory exists -

/usr/hdp/current/zookeeper-client/bin/zkCli.sh -server x.server.com,y.server.com,z.server.com get /
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

Check the HBase master log for additional information about the ZooKeeper Kerberos login. You should see information shortly after the process starts which prints the ticket lifetime information. There may be other exceptions in the log about failure to login to Kerberos that result in this znode creation failing.

View solution in original post

4 REPLIES 4
Highlighted

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

Check the HBase master log for additional information about the ZooKeeper Kerberos login. You should see information shortly after the process starts which prints the ticket lifetime information. There may be other exceptions in the log about failure to login to Kerberos that result in this znode creation failing.

View solution in original post

Highlighted

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

Thanks @Josh Elser

I analyzed the issue further, and found that the problem in Zookeeper SASL. After kerberos, Zookeeper is expecting the port number 2888-3888 to be opened between all the 3 Zookeper servers. However, I hadn't opened that range of ports. Hence SASL error was thrown even with a simple ./zkCli.sh command. I have asked the customer to open the port range.

Please let me know if this is not correct.

Regards,

Highlighted

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

Your terminology is off, but the explanation seems plausible :). 2888-3888 is the range used by ZK internal communication (ZK servers talking to each other). I can imagine that if ZK servers couldn't communicate with each other, ZK would not operate as expected. SASL is just way of performing authentication and has nothing to do with the low-level transport over the wire.

Highlighted

Re: After Kerberos, HBase doesn't start due to AuthFailed for /hbase-secure

Thank you @Josh Elser :)

Don't have an account?
Coming from Hortonworks? Activate your account here