Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hbase active and standby shutting down automatically

Solved Go to solution

Hbase active and standby shutting down automatically

Expert Contributor

Hi Folks,

I am using HDP 2.6.4 version, where i have installed hbase active and standby server with three region server.

I have started both on after another and both started successfully but both node automatically shutting down in some down while our HDFS is fully healthy. Following are the logs from hbase:

2018-12-10 10:17:20,767 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=417.42 KB, freeSize=395.89 MB, m ax=396.30 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions =209, evicted=0, evictedPerRun=0.0 2018-12-10 10:18:48,539 INFO [Init Namespace Manager and Quota Manager Async] master.TableNamespaceManager: Waiting for n amespace table to be online. Time waited = 2100006 ms. 2018-12-10 10:22:20,767 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=417.42 KB, freeSize=395.89 MB, m ax=396.30 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions =239, evicted=0, evictedPerRun=0.0 2018-12-10 10:23:48,538 ERROR [Init Namespace Manager and Quota Manager Async] master.HMaster: Namespace manager failed to start. java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:109) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1162) at java.lang.Thread.run(Thread.java:745) 2018-12-10 10:23:48,539 FATAL [Init Namespace Manager and Quota Manager Async] master.HMaster: Master server abort: loaded coprocessors are: [] 2018-12-10 10:23:48,539 FATAL [Init Namespace Manager and Quota Manager Async] master.HMaster: Shutdown Master due to name space manager failed to start. java.io.IOException: Timedout 2400000ms waiting for namespace table to be assigned at org.apache.hadoop.hbase.master.TableNamespaceManager.start(TableNamespaceManager.java:109) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1162) at java.lang.Thread.run(Thread.java:745) 2018-12-10 10:23:48,539 INFO [Init Namespace Manager and Quota Manager Async] regionserver.HRegionServer: STOPPED: Shutdo wn Master due to namespace manager failed to start.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Hbase active and standby shutting down automatically

Expert Contributor

Hi @Vinay,

It's seems to be an issue with stale WAL splits, try removing WAL's from below hdfs location and restart the Hbase and region services.

/apps/hbase/data/WALs/ 

Please note that, removing them is almost never an ideal situation unless there is no data in HBase.

2 REPLIES 2

Re: Hbase active and standby shutting down automatically

Expert Contributor

Hi @Vinay,

It's seems to be an issue with stale WAL splits, try removing WAL's from below hdfs location and restart the Hbase and region services.

/apps/hbase/data/WALs/ 

Please note that, removing them is almost never an ideal situation unless there is no data in HBase.

Re: Hbase active and standby shutting down automatically

Expert Contributor
@VinayPlease login and accept the answer if you find this helpful. Thanks
Don't have an account?
Coming from Hortonworks? Activate your account here