Created 04-23-2018 10:28 PM
I lost a drive on one of my data nodes that apparently stored some ambari-metrics collection stuff. Besides the hadoop directory structure that was created when the datanode was restarted I see a var/lib/ambari-metrics-collector that was created as well
replaced the drive and everything is back up except the metrics collector. Getting these errors
22:22:12,598 WARN [main-SendThread(datanode03.foo:61181)] ClientCnxn:1146 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused
22:22:04,011 WARN [main] DefaultPhoenixDataSource:84 - Unable to connect to HBase store using Phoenix. java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
Created 04-23-2018 11:59 PM
@Jon Page
As we see the error as following:
22:22:12,598 WARN [main-SendThread(datanode03.foo:61181)] ClientCnxn:1146 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused 22:22:04,011 WARN [main] DefaultPhoenixDataSource:84 - Unable to connect to HBase store using Phoenix. java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations
So can you please try cleaning up Zookeeper state and then try to restart AMS collector again. This kind of messages can be seen usually because of improper shutdown or losing of the datastore (DataNode).
Please check the property "hbase.tmp.dir" inside the "Advanced ams-hbase-site" in ambari UI. Usually the default value for this property is "/var/lib/ambari-metrics-collector/hbase-tmp/" Please try to clean that directory as well before restarting the AMS.
# rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper
.
More details about the AMS Cleanup can be found here (It is not needed to Clean everything for now) Just try to cleanup the zookeeper "hbase.zookeeper.property.dataDir" as mentioned in : https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data