Hbase active and standby shutting down automatically

[#1]

Hi Folks,

I am using HDP 2.6.4 version, where i have installed hbase active and standby server with three region server.

I have started both on after another and both started successfully but both node automatically shutting down in some down while our HDFS is fully healthy. Following are the logs from hbase:

2018-12-10 10:17:20,767 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=417.42 KB, freeSize=395.89 MB, m ax=396.30 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions =209, evicted=0, evictedPerRun=0.0 2018-12-10 10:18:48,539 INFO [Init Namespace Manager and Quota Manager Async] master.TableNamespaceManager: Waiting for n amespace table to be online. Time waited = 2100006 ms. 2018-12-10 10:22:20,767 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=417.42 KB, freeSize=395.89 MB, m ax=396.30 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions =239, evicted=0, evictedPerRun=0.0 2018-12-10 10:23:48,538 ERROR [Init Namespace Manager and Quota Manager Async] master.HMaster: Namespace manager failed to start. java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:109) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1162) at java.lang.Thread.run(Thread.java:745) 2018-12-10 10:23:48,539 FATAL [Init Namespace Manager and Quota Manager Async] master.HMaster: Master server abort: loaded coprocessors are: [] 2018-12-10 10:23:48,539 FATAL [Init Namespace Manager and Quota Manager Async] master.HMaster: Shutdown Master due to name space manager failed to start. java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:109) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1162) at java.lang.Thread.run(Thread.java:745) 2018-12-10 10:23:48,539 INFO [Init Namespace Manager and Quota Manager Async] regionserver.HRegionServer: STOPPED: Shutdo wn Master due to namespace manager failed to start.

[#2 · Accepted solution]

Hi @Vinay,

It's seems to be an issue with stale WAL splits, try removing WAL's from below hdfs location and restart the Hbase and region services.

/apps/hbase/data/WALs/ 

Please note that, removing them is almost never an ideal situation unless there is no data in HBase.

[#3]

@VinayPlease login and accept the answer if you find this helpful. Thanks