Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HBase Master stops in Kerberized cluster after : Timedout 2400000ms waiting for namespace table to be assigned

Highlighted

HBase Master stops in Kerberized cluster after : Timedout 2400000ms waiting for namespace table to be assigned

New Contributor

I have a Kerberized cluster (HDP version: 2.4.3 and Ambari version: 2.2.2.0) in which HBase service is installed. But, the HBase master stops after timeout with following error:

hbase master log:

2017-10-27 12:56:56,437 ERROR [Thread-100] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process. 2017-10-27 13:11:56,437 ERROR [Thread-100] master.HMaster: Master failed to complete initialization after 900000ms. Please consider submitting a bug report including a thread dump of this process. 2017-10-27 13:22:23,320 FATAL [hostname:60000.activeMasterManager] master.HMaster: Failed to become active master java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1015) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:809) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793) at java.lang.Thread.run(Thread.java:748) 2017-10-27 13:22:23,321 FATAL [hostname:60000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: [org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor] 2017-10-27 13:22:23,321 FATAL [hostname:60000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown. java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:104) at org.apache.hadoop.hbase.master.HMaster.initNamespace(HMaster.java:1015) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:809) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793) at java.lang.Thread.run(Thread.java:748)

regionserver log:

2017-10-27 13:30:39,787 ERROR [RS_OPEN_PRIORITY_REGION-hostname:16020-1] handler.OpenRegionHandler: Failed open of region=hbase:namespace,,1508913064554.16ee288e7e2f92b959283a91a2205c93., starting to roll back the global memstore size. org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'hbase/principal' (action=admin) at org.apache.ranger.authorization.hbase.AuthorizationSession.publishResults(AuthorizationSession.java:254) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.authorizeAccess(RangerAuthorizationCoprocessor.java:595) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.requirePermission(RangerAuthorizationCoprocessor.java:664) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preOpen(RangerAuthorizationCoprocessor.java:872) at org.apache.ranger.authorization.hbase.RangerAuthorizationCoprocessor.preOpen(RangerAuthorizationCoprocessor.java:778) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$1.call(RegionCoprocessorHost.java:430) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1748) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705) at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preOpen(RegionCoprocessorHost.java:426) at org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:812) at org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:796) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6356) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6317) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6288) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6244) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:6195) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:362) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:129) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

I have tried by seting following configurations :

hbase.master.namespace.init.timeout = 2400000
hbase.regionserver.executor.openregion.threads = 200

But, after 40 minutes the master goes down. Is there something that I am missing?

Thanks in advance.

1 REPLY 1

Re: HBase Master stops in Kerberized cluster after : Timedout 2400000ms waiting for namespace table to be assigned

Expert Contributor

Hi @Pooja Kamle

following error indicates that hbase user does not have correct permissions. Looks like you have ranger enabled. Can you please check if you have correct privileges for hbase user.

ERROR [RS_OPEN_PRIORITY_REGION-hostname:16020-1] handler.OpenRegionHandler: Failed open of region=hbase:namespace,,1508913064554.16ee288e7e2f92b959283a91a2205c93., starting to roll back the global memstore size. org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient permissions for user 'hbase/principal' (action=admin

Also make sure following property is set correctly:

Ambari Web UI > hbase > config > Advanced hbase-site >hbase.superuser=hbase

For Ranger Policy: Sample Screenshot

40045-screen-shot-2017-10-30-at-22517-pm.png