Hbase active and standby shutting down automatically

vinayk — Mon, 10 Dec 2018 19:49:30 GMT

Hi Folks,

I am using HDP 2.6.4 version, where i have installed hbase active and standby server with three region server.

I have started both on after another and both started successfully but both node automatically shutting down in some down while our HDFS is fully healthy. Following are the logs from hbase:

2018-12-10 10:17:20,767 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=417.42 KB, freeSize=395.89 MB, m ax=396.30 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions =209, evicted=0, evictedPerRun=0.0 2018-12-10 10:18:48,539 INFO [Init Namespace Manager and Quota Manager Async] master.TableNamespaceManager: Waiting for n amespace table to be online. Time waited = 2100006 ms. 2018-12-10 10:22:20,767 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=417.42 KB, freeSize=395.89 MB, m ax=396.30 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions =239, evicted=0, evictedPerRun=0.0 2018-12-10 10:23:48,538 ERROR [Init Namespace Manager and Quota Manager Async] master.HMaster: Namespace manager failed to start. java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:109) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1162) at java.lang.Thread.run(Thread.java:745) 2018-12-10 10:23:48,539 FATAL [Init Namespace Manager and Quota Manager Async] master.HMaster: Master server abort: loaded coprocessors are: [] 2018-12-10 10:23:48,539 FATAL [Init Namespace Manager and Quota Manager Async] master.HMaster: Shutdown Master due to name space manager failed to start. java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:109) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1162) at java.lang.Thread.run(Thread.java:745) 2018-12-10 10:23:48,539 INFO [Init Namespace Manager and Quota Manager Async] regionserver.HRegionServer: STOPPED: Shutdo wn Master due to namespace manager failed to start.

Re: Hbase active and standby shutting down automatically

sampathkumar_ma — Mon, 10 Dec 2018 21:03:03 GMT

Hi @Vinay,

It's seems to be an issue with stale WAL splits, try removing WAL's from below hdfs location and restart the Hbase and region services.

/apps/hbase/data/WALs/

Please note that, removing them is almost never an ideal situation unless there is no data in HBase.

Re: Hbase active and standby shutting down automatically

sampathkumar_ma — Wed, 12 Dec 2018 18:45:06 GMT

@VinayPlease login and accept the answer if you find this helpful. Thanks

question Re: Hbase active and standby shutting down automatically in Archives of Support Questions (Read Only)

Hbase active and standby shutting down automatically

Re: Hbase active and standby shutting down automatically

Re: Hbase active and standby shutting down automatically