KeeperErrorCode = ConnectionLoss for /hbase/master

[#1]

errorlog.txtThis thread is related to my post related to Metrics collector not starting (https://community.hortonworks.com/questions/23512/metrics-collector-is-not-starting-showing-error-re.html). I thought of creating a new thread to highlight the main issue that I see in the log files. In general, Metric Collector is not starting. The logs have many information related to the error. The error that is mainly shown

exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master

In the same log file, I also see the following line

zookeeper.ClientCnxn: Opening socket connection to server localhost/0:0:0:0:0:0:0:1:61181.

If 'localhost' is the issue here, then I am not sure why localhost is used here instead of FQDN and I do not know which configuration file will help me to use FQN instead of localhost.

I have read many threads related to this error but none of them helped me fix this issue. Can anyone please help me understand this issue.

I am pasting the value some of the properties:

hbase.rootdir = hdfs://item-70288:8020/apps/hbase/data

hbase.cluster.distributed= true

Metrics service operation mode = embedded

hbase.zookeeper.property.clientPort= 2181

hbase.zookeeper.quorum =item-70288

Update #1

=========

The command netstat -anp |grep 61181 is not returning anything. It seems nothing is listening on 61181.

I am attaching the full error log with this post.

[#2]

I have fixed the issue related to Metrics Monitor not starting. Even though I have no idea why the above error came, I fixed by changing/setting the following property under Ambari Metrics Monitor configuration through Ambari.

1. set hbase.zookeeper.property.clientPort to 2181. I had observed that this was set to 61181, which I am not sure how it happend. This is a very important property. The Zookeeper server also runs at the same port, so I guess the hbase.zookeeper.property.clientPort should also be set to the same port. If these ports are different, then there will be a number of errors thrown by Ambari-metrics collector service is started. Therefore make sure this property is set to the same port as the zookeeper server.
2. Change the "Metrics Service operation mode" to distributed.
3. Set hbase.cluster.distributed to true
4. Set hbase.rootdir to a hdfs folder. I created a new folder in the hdfs root and made hdfs the owner of the folder.

After I did this, I restarted only the Ambari Metric collector and this time it was happy to start with a green flag 😉

[#3]

I had a same issue yesterday, and it was fixed by changing mode from embedded to distributed as mentioned by @Pradeep kumar, except I did not change the hbase.zookeeper.property.clientPort to 2181.

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/ams_collec...

[#4]

Hi, I'm facing the same issue as well. This is my hbase-site.xml
<configuration>
<property>
<name>hbase.master</name>
<value>nn:9000</value>
</property>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:9000/hbase</value>
</property>
<property>
<name>hbase.master.port</name>
<value>60001</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>nn,dd1,dd2,dd3</value>
</property>
<property>
<name>hbase.zookeeper.property.maxClientCnxns</name>
<value>35</value>
</property>
<property>
<name>hbase.zookeeper.znode.parent</name>
<value>/hbase</value>
</property>
</configuration>

Please help me out!