ambari metrics collector won't start

[#1]

I lost a drive on one of my data nodes that apparently stored some ambari-metrics collection stuff. Besides the hadoop directory structure that was created when the datanode was restarted I see a var/lib/ambari-metrics-collector that was created as well

replaced the drive and everything is back up except the metrics collector. Getting these errors

22:22:12,598 WARN [main-SendThread(datanode03.foo:61181)] ClientCnxn:1146 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused

22:22:04,011 WARN [main] DefaultPhoenixDataSource:84 - Unable to connect to HBase store using Phoenix. java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations

[#2]

@Jon Page
As we see the error as following:

22:22:12,598 WARN [main-SendThread(datanode03.foo:61181)] ClientCnxn:1146 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: 
Connection refused
22:22:04,011 WARN [main] DefaultPhoenixDataSource:84 - Unable to connect to HBase store using 
Phoenix. java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations

So can you please try cleaning up Zookeeper state and then try to restart AMS collector again. This kind of messages can be seen usually because of improper shutdown or losing of the datastore (DataNode).

Please check the property "hbase.tmp.dir" inside the "Advanced ams-hbase-site" in ambari UI. Usually the default value for this property is "/var/lib/ambari-metrics-collector/hbase-tmp/" Please try to clean that directory as well before restarting the AMS.

# rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper

.

More details about the AMS Cleanup can be found here (It is not needed to Clean everything for now) Just try to cleanup the zookeeper "hbase.zookeeper.property.dataDir" as mentioned in : https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data