question Re: Ambari metrics collector -unble to restart in Archives of Support Questions (Read Only) https://community.cloudera.com/t5/Archives-of-Support-Questions/Ambari-metrics-collector-unble-to-restart/m-p/136098#M27572 <P>Thanks. It is working. </P> Sun, 08 May 2016 02:51:12 GMT anand_ranganath 2016-05-08T02:51:12Z Ambari metrics collector -unble to restart https://community.cloudera.com/t5/Archives-of-Support-Questions/Ambari-metrics-collector-unble-to-restart/m-p/136096#M27570 <P>Ambari metrics collector was working fine for 6 months and suddenly it stopped working. This is the error we are getting.</P><UL> <LI>hbase.rootdir - /mnt/data/ambari-metrics-collector/hbase</LI><LI>hbase.cluster.distributed -false</LI><LI>Metrics service operation mode -embedded</LI><LI>hbase.zookeeper.property.clientPort - 61181</LI></UL><UL><LI> 06:19:30,678 WARN [main] RecoverableZooKeeper:253 - Possibly transient ZooKeeper, quorum=localhost:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connect ionLoss for /hbase 06:19:30,678 ERROR [main] RecoverableZooKeeper:255 - ZooKeeper exists failed after 4 attempts 06:19:30,679 WARN [main] ZKUtil:484 - hconnection-0xd78795, quorum=localhost:61181, baseZNode=/hbase Unable to set watcher on znode (/hbase) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:199) at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:874) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$600(ConnectionManager.java:585) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1553) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1599) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1653) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1860) at org.apache.hadoop.hbase.client.HBaseAdmin$MasterCallable.prepare(HBaseAdmin.java:3363) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:125) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3390) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:408) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:429) at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:762) at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1107) at org.apache.phoenix.query.DelegateConnectionQueryServices.createTable(DelegateConnectionQueryServices.java:110) at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:1527) at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:535) at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:184) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:260) at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:252) at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53) at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:250) at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1026) at org.apache.phoenix.query.ConnectionQueryServicesImpl$9.call(ConnectionQueryServicesImpl.java:1532)</LI></UL> Sat, 07 May 2016 13:46:46 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Ambari-metrics-collector-unble-to-restart/m-p/136096#M27570 anand_ranganath 2016-05-07T13:46:46Z Re: Ambari metrics collector -unble to restart https://community.cloudera.com/t5/Archives-of-Support-Questions/Ambari-metrics-collector-unble-to-restart/m-p/136097#M27571 <P>Trying cleaning up Zookeeper state and restarting, rarely due to improper shutdown in embedded mode the state gets corrupted:</P><P>Check value of hbase.tmp.dir, default = /var/lib/ambari-metrics-collector/hbase-tmp/</P><P>rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/version-2/*</P><P>Restart service.</P> Sun, 08 May 2016 02:13:17 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Ambari-metrics-collector-unble-to-restart/m-p/136097#M27571 sidwagle 2016-05-08T02:13:17Z Re: Ambari metrics collector -unble to restart https://community.cloudera.com/t5/Archives-of-Support-Questions/Ambari-metrics-collector-unble-to-restart/m-p/136098#M27572 <P>Thanks. It is working. </P> Sun, 08 May 2016 02:51:12 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Ambari-metrics-collector-unble-to-restart/m-p/136098#M27572 anand_ranganath 2016-05-08T02:51:12Z