Community Articles

Find and share helpful community-sourced technical articles.
Labels (1)
avatar

Sometimes Ambari Metrics stops displaying data in the dashboards.

Or You may be getting time out issues that were not solved (see

https://community.hortonworks.com/questions/4726/ambari-metric-collector-error-sending-metric-to-se....)

At other times, especially on a new install you may receive this error in the Collector Log

2016-01-25 15:33:31,195 WARN 
org.apache.hadoop.yarn.server.applicationhistoryservice.metrics.timeline.query.DefaultPhoenixDataSource:
 Unable to connect to HBase store using Phoenix.
org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=SYSTEM.CATALOG.

This is usually due to AMS Data being corrupt.

  1. Shut down Ambari Monitors, and Collector via Ambari
  2. Cleared out the /var/lib/ambari-metrics-collector dir for fresh restart
  3. From Ambari -> Ambari Metrics -> Config -> Advanced ams-hbase-site get the hbase.rootdir and hbase-tmp directory

    1583-screen-shot-2016-01-26-at-124326-pm.png

    1584-screen-shot-2016-01-26-at-125218-pm.png

  4. Delete or Move the hbase-tmp and hbase.rootdir directories to an archive folder
  5. Started AMS.
  6. All services will came online and graphs started to display, after a few minutes
16,755 Views
Comments
avatar
New Contributor

I hereby declare my undying love for you for fixing the error I've found this morning:

Connection failed: [Errno 111] Connection refused to 0.0.0.0:6188

If I could afford it, I would by you a lifelong supply of the beverage of your choice - but I can't, so thanks shall have to suffice 🙂

avatar
Contributor

Thanks. This helped me purge metrics too as Ambari metrics services were stuck because of a disk full error. I also updated metrics retention parameters.

avatar
Contributor

I got the Ambari Metrics error "Connection failed: [Errno 111] Connection refused to metrics02-node:6188" from Ambari UI.

Ambari metrics collector connection refused : metrics data not available on Ambari Dashboard

Alert : Metrics Collector Process. Connection failed: [Errno 111] Connection refused to metrics02-node:6188

As you mentioned, i removed data from hbase.tmp.dir and hbase.rootdir.

rm -rf /var/lib/ambari-metrics-collector/hbase/*

rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/*

Later restared Ambari Metrics from Ambari. It worked fine. Thanks a lot.

avatar
Rising Star

Thanks, it's works.