How to solve Ambari Metrics corrupted data

[#1]

Sometimes Ambari Metrics stops displaying data in the dashboards.

Or You may be getting time out issues that were not solved (see

https://community.hortonworks.com/questions/4726/ambari-metric-collector-error-sending-metric-to-se....)

At other times, especially on a new install you may receive this error in the Collector Log

2016-01-25 15:33:31,195 WARN 
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource:
 Unable to connect to HBase store using Phoenix.
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG.

This is usually due to AMS Data being corrupt.

1. Shut down Ambari Monitors, and Collector via Ambari
2. Cleared out the /var/lib/ambari-metrics-collector dir for fresh restart
3. From Ambari -> Ambari Metrics -> Config -> Advanced ams-hbase-site get the hbase.rootdir and hbase-tmp directory

   

   

4. Delete or Move the hbase-tmp and hbase.rootdir directories to an archive folder
5. Started AMS.
6. All services will came online and graphs started to display, after a few minutes

[#2]

I hereby declare my undying love for you for fixing the error I've found this morning:

Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

If I could afford it, I would by you a lifelong supply of the beverage of your choice - but I can't, so thanks shall have to suffice 🙂

[#3]

Thanks. This helped me purge metrics too as Ambari metrics services were stuck because of a disk full error. I also updated metrics retention parameters.

[#4]

I got the Ambari Metrics error "Connection failed: [Errno 111] Connection refused to metrics02-node:6188" from Ambari UI.

Ambari metrics collector connection refused : metrics data not available on Ambari Dashboard

Alert : Metrics Collector Process. Connection failed: [Errno 111] Connection refused to metrics02-node:6188

As you mentioned, i removed data from hbase.tmp.dir and hbase.rootdir.

rm -rf /var/lib/ambari-metrics-collector/hbase/*

rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/*

Later restared Ambari Metrics from Ambari. It worked fine. Thanks a lot.

[#5]

Thanks, it's works.