HBase Master dying after startup

[#1]

Greetings, all.

I have a very simple configuration: 3 node cluster -- one master, two workers; HDFS NN on master, HDFS SNN on one of the workers, and HDFS DN on each node. ZooKeeper successfully started on all three nodes with an established quorum and elected leader, etc. When I attempt to start HBase Master on the master node, it waits while HDFS NN works in "safe mode", doing its block replication thing. (While using the HBase shell to issue a status command, you get "HBase Initializing", etc.). Eventually, HBase Master goes away, because it times out waiting for HDFS NN to finish and exist safe mode. I have three questions:

1) Does my (simplest possible attempt) configuration, as described above, seem reasonable?

2) I'd prefer not to force HDFS out of safe mode; but as this is just an initial startup of a freshly provisioned Hadoop cluster, I assume that might not cause any issues. Do you agree?

3) Is there a threshold setting somewhere in the config. files that I can raise to ensure HBase Master waits longer for HDFS? I've been searching, but haven't found anything yet.

Attached is an abbreviated screen cap of log output...some messages just before the initial warning, and then the exception indicating that HBase Master has finally shutdown.

Thanks in advance for your help on this one!

~ Joseph

[#2 · Accepted solution]

@Joseph Hawkins

1. There is nothing inherently wrong with your cluster layout for a small cluster.

2. You generally don't want to force HDFS out of safe mode. Can you determine why HDFS is in safe mode? If it is initial configuration, there shouldn't be any good reasons for it to be in safe mode.

3. I haven't found a configuration setting for HBase to wait for HDFS. The assumption is that HDFS is running properly and writable. Safe mode makes HDFS read-only which is why HBase shutsdown.

How are you starting the processes? If you are doing the process start up from the command line, then you do need to ensure that HDFS comes up prior to start HBase.

Is this cluster managed with Ambari? When you start services with Ambari, it will start the services in proper sequence and ensure the components are up before starting the next one.

[#3]

It happens when master is getting timeout while initializing the namespace table. (As far I know)

Increase the value for hbase.master.namespace.init.timeout in hbase-site.xml