I have a Kerberized cluster (HDP version: 2.4.3 and Ambari version: 2.2.2.0) in which HBase service is installed. But, the HBase master stops after timeout with following error:
hbase master log:
2017-10-27 12:56:56,437 ERROR [Thread-100] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.
2017-10-27 13:11:56,437 ERROR [Thread-100] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process.
2017-10-27 13:22:23,320 FATAL [hostname:60000.activeMasterManager] master.HMaster: Failed to become active master
java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1015)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:809)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793)
at java.lang.Thread.run(Thread.java:748)
2017-10-27 13:22:23,321 FATAL [hostname:60000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor]
2017-10-27 13:22:23,321 FATAL [hostname:60000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned
at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104)
at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1015)
at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:809)
at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193)
at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793)
at java.lang.Thread.run(Thread.java:748)
regionserver log:
2017-10-27 13:30:39,787 ERROR [RS_OPEN_PRIORITY_REGION-hostname:16020-1] handler.OpenRegionHandler: Failed open of region=hbase:namespace,,1508913064554.16ee288e7e2f92b959283a91a2205c93., starting to roll back the global memstore size.
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'hbase/principal' (action=admin)
at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:254)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:595)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuthorizationCoprocessor.java:664)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preOpen(RangerAuthorizationCoprocessor.java:872)
at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preOpen(RangerAuthorizationCoprocessor.java:778)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$1.call(RegionCoprocessorHost.java:430)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preOpen(RegionCoprocessorHost.java:426)
at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:812)
at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:796)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6356)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6317)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6288)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6244)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6195)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
I have tried by seting following configurations :
hbase.master.namespace.init.timeout = 2400000
hbase.regionserver.executor.openregion.threads = 200
But, after 40 minutes the master goes down. Is there something that I am missing?
Thanks in advance.
