Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ambari metrics collector won't start

Highlighted

ambari metrics collector won't start

Contributor

I lost a drive on one of my data nodes that apparently stored some ambari-metrics collection stuff. Besides the hadoop directory structure that was created when the datanode was restarted I see a var/lib/ambari-metrics-collector that was created as well

replaced the drive and everything is back up except the metrics collector. Getting these errors

22:22:12,598 WARN [main-SendThread(datanode03.foo:61181)] ClientCnxn:1146 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused

22:22:04,011 WARN [main] DefaultPhoenixDataSource:84 - Unable to connect to HBase store using Phoenix. java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations

1 REPLY 1

Re: ambari metrics collector won't start

Super Mentor

@Jon Page
As we see the error as following:

22:22:12,598 WARN [main-SendThread(datanode03.foo:61181)] ClientCnxn:1146 - Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: 
Connection refused
22:22:04,011 WARN [main] DefaultPhoenixDataSource:84 - Unable to connect to HBase store using 
Phoenix. java.sql.SQLException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the locations


So can you please try cleaning up Zookeeper state and then try to restart AMS collector again. This kind of messages can be seen usually because of improper shutdown or losing of the datastore (DataNode).

Please check the property "hbase.tmp.dir" inside the "Advanced ams-hbase-site" in ambari UI. Usually the default value for this property is "/var/lib/ambari-metrics-collector/hbase-tmp/" Please try to clean that directory as well before restarting the AMS.

# rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper

.

More details about the AMS Cleanup can be found here (It is not needed to Clean everything for now) Just try to cleanup the zookeeper "hbase.zookeeper.property.dataDir" as mentioned in : https://cwiki.apache.org/confluence/display/AMBARI/Cleaning+up+Ambari+Metrics+System+Data

Don't have an account?
Coming from Hortonworks? Activate your account here