Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hbase.MasterNotRunningException

SOLVED Go to solution

hbase.MasterNotRunningException

New Contributor

Seeing this error on a 8 node cluster. Able to connect from the node where zookeeper is deployed.

 

2013-10-18 17:01:08,656 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: getMaster attempt 0 of 10 failed; retrying after sleep of 1009
org.apache.hadoop.hbase.MasterNotRunningException
	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:706)
	at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:126)
	at org.apache.hadoop.hbase.thrift.ThriftServerRunner$HBaseHandler.<init>(ThriftServerRunner.java:513)
	at org.apache.hadoop.hbase.thrift.ThriftServerRunner.<init>(ThriftServerRunner.java:228)
	at org.apache.hadoop.hbase.thrift.ThriftServer.doMain(ThriftServer.java:100)
	at org.apache.hadoop.hbase.thrift.ThriftServer.main(ThriftServer.java:237)
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: hbase.MasterNotRunningException

New Contributor

We got help from support to solve this. Here the solution. Solution is temporary for that hive session.

 

hive> set hbase.zookeeper.quorum=list of host names

 

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.4.0.jar

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/guava-11.0.2.jar;

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/zookeeper.jar;

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hbase/hbase-0.94.6-cdh4.4.0-security.jar;

6 REPLIES 6

Re: hbase.MasterNotRunningException

Master Collaborator

Can you give a bit more detail as to what you are doing when you encounter this error?  And is the machine where you are seeing this one of those 8 nodes in the cluster?  Or an external machine?

 

I've seen this before when a client app outside the cluster was unable to connect to the zookeeper quorum because a local copy of the hbase-site.xml file was not in the application's path therefore it did not know who the zookeeper servers were and the error looks like yours.  The property that needs to be specified for the client is: hbase.zookeeeper.quorum.

 

http://hbase.apache.org/book/zookeeper.html

Re: hbase.MasterNotRunningException

New Contributor

It is one of the 8 nodes. We see the same issue when running Hive and Impala queries from Hue on the node where Zookeeper is not running. 

 

So we have Zookeeper on Nodes 3,4,5. Hue is running on Node 1. We see this issue executing Hive query in Hue.

Re: hbase.MasterNotRunningException

Master Collaborator

do you have the zookeeper.quorum property I mentioned previously in your /etc/hbase/conf/hbase-site.xml file on these systems?  It sounds like your hbase clients  (any app trying to access the HBase service) are trying to use the default property for the ZK quorum, which would have them looking on the localhost for a ZK server.  This is why it works on nodes that are running a ZK instance.  You need a valid hbase-site.xml file on each node that specifies the ZK quorum.  It was described in that link I posted.  I hope that helps.

Re: hbase.MasterNotRunningException

New Contributor

thanks for your response. On each node we have the following in hbase-site.xml. I changed the domain to zzz here

 

<property>
<name>hbase.zookeeper.quorum</name>
<value>zzz1.zzz.com,zzz2.zzz.com,zzz3.zzz.com</value>
</property>

Re: hbase.MasterNotRunningException

New Contributor

I need one clarification. I see this configuration in /etc/hbase/conf/hbase-site.xml

 

<property>
<name>hbase.zookeeper.quorum</name>
<value>zzz1.zzz.com,zzz2.zzz.com,zzz3.zzz.com</value>
</property>

 

Added two more zookeepers thru cloudera manager services. when I view hbase-> instances -> region servers -> process -> show/hide configs/hbase-site.xml I see the 5 zookeeper hosts. But /etc/hbase/conf still has 3. Does Cloudera use /etc/hbase/conf/hbase-site.xml by default or does it store the updated hbase-site.xml some where else?

Highlighted

Re: hbase.MasterNotRunningException

New Contributor

We got help from support to solve this. Here the solution. Solution is temporary for that hive session.

 

hive> set hbase.zookeeper.quorum=list of host names

 

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.4.0.jar

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/guava-11.0.2.jar;

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hive/lib/zookeeper.jar;

hive> ADD JAR /opt/cloudera/parcels/CDH-4.4.0-1.cdh4.4.0.p0.39/lib/hbase/hbase-0.94.6-cdh4.4.0-security.jar;