@Aravindan Vijayan , when fire the following, i don't see any output
[root@sandbox hbase-tmp]# netstat -a | grep 61181
seems the local zookeeper is not running .. how to start this ?
@Aravindan Vijayan - any updates on this ?
I tried re-installing AMBARI METRICS, and still see the same error..
Pls note - Grafana & Metrics Monitor are starting up fine.
2016-12-11 17:00:38,348 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=localhost:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-secure 2016-12-11 17:00:38,348 ERROR org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: ZooKeeper exists failed after 4 attempts 2016-12-11 17:00:38,348 WARN org.apache.hadoop.hbase.zookeeper.ZKUtil: hconnection-0x155aa3ef-0x158eed3ab0f0002, quorum=localhost:61181, baseZNode=/ams-hbase-secure Unable to set watcher on znode (/ams-hbase-secure) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-secure at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:221) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:417)
I see that HBase is trying to connect to zk at ping localhost.localdomain. Metrics collector is still trying to connect to localhost. Is there a way you can change ur settings such that it tries to connect localhost.?
I guess it is taking it from /etc/hosts, not sure how to change that -
[root@sandbox ~]# cat /etc/hosts # File is generated from /usr/lib/hue/tools/start_scripts/gen_hosts.sh # Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 10.0.2.15 sandbox.hortonworks.com sandbox ambari.hortonworks.com
For now, i've changed it Metrics Collector to distributed mode, it seems to be working fine.
Can you check if there's a hs_err_pidXXXX.log file in /var/log/ambari-metrics-collector? I had the same issue and this log said "There is insufficient memory for the Java Runtime Environment to continue." Tuning the memory settings of Ambari Metrics according to these guidelines solved the problem for me: https://docs.hortonworks.com/HDPDocuments/Ambari-188.8.131.52/bk_ambari_reference_guide/content/_ams_gene...