Support Questions
Find answers, ask questions, and share your expertise

HBASE master and region servers stop automatically

New Contributor

1)Kerberos enabled

2)Ranger enabled

3)/var/log/hbase/hbase-hbase-master-<DN>.log

2018-08-23 17:19:06,408 INFO [main-SendThread(<DN>:2181)] zookeeper.Login: successfully logged in. 2018-08-23 17:19:06,409 INFO [Thread-10] zookeeper.Login: TGT refresh thread started. 2018-08-23 17:19:06,416 INFO [main-SendThread(FQDN:2181)] client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. 2018-08-23 17:19:06,420 INFO [Thread-10] zookeeper.Login: TGT valid starting at: Thu Aug 23 17:19:06 IST 2018 2018-08-23 17:19:06,420 INFO [Thread-10] zookeeper.Login: TGT expires: Fri Aug 24 17:19:06 IST 2018 2018-08-23 17:19:06,420 INFO [Thread-10] zookeeper.Login: TGT refresh sleeping until: Fri Aug 24 13:29:07 IST 2018 2018-08-23 17:19:06,423 INFO [main-SendThread(FQDN:2181)] zookeeper.ClientCnxn: Opening socket connection to server FQDN/10.21.58.13:2181. Will attempt to SASL-authenticate using Login Context section 'Client' 2018-08-23 17:19:06,432 INFO [main-SendThread(FQDN:2181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.21.58.12:46578, server: FQDN/10.21.58.13:2181 2018-08-23 17:19:06,443 INFO [main-SendThread(FQDN:2181)] zookeeper.ClientCnxn: Session establishment complete on server FQDN/10.21.58.13:2181, sessionid = 0x3656691264d000c, negotiated timeout = 60000 2018-08-23 17:19:06,468 ERROR [main-SendThread(FQDN:2181)] client.ZooKeeperSaslClient: SASL authentication failed using login context 'Client'. 2018-08-23 17:19:06,576 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2886) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:235) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2899) Caused by: org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /hbase-secure at org.apache.zookeeper.KeeperException.create(KeeperException.java:123) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.createNonSequential(RecoverableZooKeeper.java:565) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.create(RecoverableZooKeeper.java:544) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1204) at org.apache.hadoop.hbase.zookeeper.ZKUtil.createWithParents(ZKUtil.java:1182) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.createBaseZNodes(ZooKeeperWatcher.java:186) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:179) at org.apache.hadoop.hbase.regionserver.HRegionServer.<init>(HRegionServer.java:617) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:442) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2880)

1 REPLY 1

Super Collaborator

can you do a klist on the shell with the same user that starts the HBase master (and post the result here).

I am a little curios on the point that the TGT expiry timestamp is the exact time of your start. Are you using a ticket or a keytab for Kerberos? And which is the configured principal for HBase?

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.