HBASE master fails to start with InvalidACL for /hbase-secure2

[#1]

Hi,

I have upgraded Ambari recently to 2.2.2.

After upgrade, Ambarai asked for restart of most of the components. I have restarted them but Hbase is restarted ahead of Zookeeper.

Now, Hbase master fails to start with below error

ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster

Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hbase-secure

Please help me on this.

Thanks,

Venkat

[#2 · Accepted solution]

I restarted all services from Ambari and Hbase came up successfully without any problem. So, it clearly shows teh problem was because of not follwoing the order while restarting the services.

But it would be great if someone let know the root cause for the above problem.

Thanks in advance

[#3]

Can you pastebin snippet of master log / zookeeper logs ?

Thanks

[#4]

Hbase master log

lp.bcbsa.com:2181 sessionTimeout=30000 watcher=master:160000x0, quorum=mdcthdpdas10lp.bcbsa.com:2181, baseZNode=/hbase-secure1 2016-05-26 16:11:43,494 WARN [main-SendThread(mdcthdpdas10lp.bcbsa.com:2181)] client.ZooKeeperSaslClient: Could not login: the client is being asked for a password, but the Zookeeper client code does not currently support obtaining a password from the user. Make sure that the client is configured to use a ticket cache (using the JAAS configuration setting 'useTicketCache=true)' and restart the client. If you still get this message after that, the TGT in the ticket cache has expired and must be manually refreshed. To do so, first determine if you are using a password or a keytab. If the former, run kinit in a Unix shell in the environment of the user who is running this Zookeeper client using the command 'kinit <princ>' (where <princ> is the name of the client's Kerberos principal). If the latter, do 'kinit -k -t <keytab> <princ>' (where <princ> is the name of the Kerberos principal, and <keytab> is the location of the keytab file). After manually refreshing your cache, restart this client. If you continue to see this message after manually refreshing your cache, ensure that your KDC host's clock is in sync with this host's clock. 2016-05-26 16:11:43,499 WARN [main-SendThread(mdcthdpdas10lp.bcbsa.com:2181)] zookeeper.ClientCnxn: SASL configuration failed: javax.security.auth.login.LoginException: No password provided Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 2016-05-26 16:11:43,500 INFO [main-SendThread(mdcthdpdas10lp.bcbsa.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server mdcthdpdas10lp.bcbsa.com/10.145.4.30:2181 2016-05-26 16:11:43,503 INFO [main-SendThread(mdcthdpdas10lp.bcbsa.com:2181)] zookeeper.ClientCnxn: Socket connection established to mdcthdpdas10lp.bcbsa.com/10.145.4.30:2181, initiating session 2016-05-26 16:11:43,509 INFO [main-SendThread(mdcthdpdas10lp.bcbsa.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server mdcthdpdas10lp.bcbsa.com/10.145.4.30:2181, sessionid = 0x154a09cb3c7005a, negotiated timeout = 30000 2016-05-26 16:11:43,516 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2290) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:233) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2304) Caused by: org.apache.zookeeper.KeeperException$InvalidACLException: KeeperErrorCode = InvalidACL for /hbase-secure1 at org.apache.zookeeper.KeeperException.create(KeeperException.java:121)

[#5]

Any update on the above error?