question Re: HBase Master dying after startup in Support Questions

HBase Master dying after startup

hawkins_house_d — Mon, 19 Aug 2019 11:14:12 GMT

Greetings, all.

I have a very simple configuration: 3 node cluster -- one master, two workers; HDFS NN on master, HDFS SNN on one of the workers, and HDFS DN on each node. ZooKeeper successfully started on all three nodes with an established quorum and elected leader, etc. When I attempt to start HBase Master on the master node, it waits while HDFS NN works in "safe mode", doing its block replication thing. (While using the HBase shell to issue a status command, you get "HBase Initializing", etc.). Eventually, HBase Master goes away, because it times out waiting for HDFS NN to finish and exist safe mode. I have three questions:

1) Does my (simplest possible attempt) configuration, as described above, seem reasonable?

2) I'd prefer not to force HDFS out of safe mode; but as this is just an initial startup of a freshly provisioned Hadoop cluster, I assume that might not cause any issues. Do you agree?

3) Is there a threshold setting somewhere in the config. files that I can raise to ensure HBase Master waits longer for HDFS? I've been searching, but haven't found anything yet.

Attached is an abbreviated screen cap of log output...some messages just before the initial warning, and then the exception indicating that HBase Master has finally shutdown.

Thanks in advance for your help on this one!

~ Joseph

Re: HBase Master dying after startup

myoung — Sat, 18 Feb 2017 00:44:50 GMT

@Joseph Hawkins

1. There is nothing inherently wrong with your cluster layout for a small cluster.

2. You generally don't want to force HDFS out of safe mode. Can you determine why HDFS is in safe mode? If it is initial configuration, there shouldn't be any good reasons for it to be in safe mode.

3. I haven't found a configuration setting for HBase to wait for HDFS. The assumption is that HDFS is running properly and writable. Safe mode makes HDFS read-only which is why HBase shutsdown.

How are you starting the processes? If you are doing the process start up from the command line, then you do need to ensure that HDFS comes up prior to start HBase.

Is this cluster managed with Ambari? When you start services with Ambari, it will start the services in proper sequence and ensure the components are up before starting the next one.

Re: HBase Master dying after startup

rchaman — Wed, 10 May 2017 05:41:51 GMT

It happens when master is getting timeout while initializing the namespace table. (As far I know)

Increase the value for hbase.master.namespace.init.timeout in hbase-site.xml

Re: HBase Master dying after startup

bigdatanoob — Mon, 28 Apr 2025 11:28:22 GMT

yas i totatally agree , but i dont get how increasing the time out fixes the problem?
the hbase master is basically not functional as long as it has not finished initializing .