metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher

mike_bronson7 — Wed, 02 Jan 2019 16:10:56 GMT

hi all

we try to start the ambari metrics and we get the following logs:

2019-01-02 06:35:16,406 ERROR org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: hconnection-0x4bdeaabb-0x1680d44c1810001, quorum=master02.sys32.com:61181, baseZNode=/ams-hbase-unsecure Received unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/table/SYSTEM.CATALOG
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
        at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:622)

and also

2019-01-02 06:35:18,256 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys32.com/101.23.98.60:61181. Will not attempt to authenticate using SASL (unknown error)
2019-01-02 06:35:18,257 WARN org.apache.zookeeper.ClientCnxn: Session 0x1680d44c1810001 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
2019-01-02 06:35:19,656 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server master02.sys32.com/101.23.98.60:61181. Will not attempt to authenticate using SASL (unknown error)

we also clear the contents of /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/* and restart Ambari metrics collector

but still metrics collector failed after few seconds

what else we can do in order to solve this issue ?

we have HDP version - 2.6.0

Re: metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher

jsensharma — Wed, 02 Jan 2019 16:19:45 GMT

@Michael Bronson

Your current Thread query looks very similar to the other HCC thread opened by you: https://community.hortonworks.com/questions/231176/metrics-collector-ams-hbase-unsecure-received-unex.html?childToView=232246#answer-232246

Can you please mark one of the HCC thread as Closed so that all the hcc users can respond to the same single thread.

Re: metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher

mike_bronson7 — Wed, 02 Jan 2019 16:46:34 GMT

@Jay Kumar SenSharma , it was because when I post the first one , then we notice that first question not appears , so I post the second one

Re: metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher

mike_bronson7 — Wed, 02 Jan 2019 16:50:23 GMT

I try also to delete this duplicate post but I cant , so I accepted your answer insted , hope this is ok

question Re: metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher in Support Questions

metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher

Re: metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher

Re: metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher

Re: metrics failed on org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher