ambari metrics not displaying completely

[#1]

Running HDP 2.5.3 and ambari 2.4.2

I am having trouble getting some of the metrics and graphs to display.

Dashboard page displays graphs OK

Service page displays graphs OK except for Ambari Metrics - graphs just show spinning circles

Main hosts pages do not show Disk Usage or Load Avg.

Specific host page sometimes (rarely) shows graphs, usually shows spinners. But if I go back to the main hosts page, disk usage and load avg will display briefly for that host, then disappear.

I have seen posts regarding the restart of ntp servers - time on all hosts are synced together. I have seen a post regarding setting up a proxy host but the cluster is on an isolated network with a local repository - no internet access.

Any ideas?

[#2]

@Jon Page

Please check if you could find any errors in Ambari metrics collector log. You could try clearing your hbase root/tmp directories if this would have caused by meta data corruption. Please note that if you remove hbase.rootdir, you could loose your historic AMS data. Please check the below URL.

https://community.hortonworks.com/questions/73577/problem-with-ambari-metrics.html

Ambari metrics monitors are used for collecting Host Metrics and sent to Ambari metrics collector. Service Metrics would be directly sent to the collector.

[#3]

@Jon Page How big is your cluster? It could be that AMS is not properly configured per cluster size or AMS HBase is writing to a contended disk. As a quick solution you could try whitelisting AMS data which reduces the write load and will give perf improvement.

https://issues.apache.org/jira/browse/AMBARI-16969

Configs: (make sure the whitelist file exists (came with the rpm))

ams-site :: timeline.metrics.whitelist.file = /etc/ambari-metrics-collector/conf/metrics_whitelist

ams-site : timeline.metrics.precision.table.durability = ASYNC_WAL

ams-site : timeline.metrics.aggregate.tables.durability = ASYNC_WAL

[#4]

43 nodes in total

I do not have the metrics_whitelist file in /etc/ambari-metrics-collector/conf

[#5]

@Jon Page

Could you try this

Logon the Ambari server clear the contents of

# /var/lib/ambari-metrics- collector/hbase-tmp/zookeeper/*

Then restart the AMS that should help

[#6]

I think my problem is with a FW. If I stop the FW on and ambari server and the ambari-metrics-collector server, things seem OK. Besides 6188, are there any other ports that AMC use?

[#7]

@Jon Page

If you are using AMS in embedded mode ZK quorum port would be 61181. But in distributed mode it is 2181 as it would be using cluster Zookeeper quorum. You might need to consider this port as well.

[#8]

running embedded mode. Added the ports, restarted iptables and ambari-metrics. Still getting spinners for graphs on the ambari-metrics page and on individual host pages, everything else seems to be OK. I haven't tried blowing away the var/lib/ambari-metrics directory

No errors in any of the logs

[#9]

The metric monitors send data to collector on 6188 port.

Did you check the log on the metric monitors? /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out

HMaster port UI port is 61310.

[#10]

@Jon Page

Which firewall are you talking of ? One of the pre-requisites is diabling the FW/TP etc did you execute those steps? Check tht these steps were all successful Prepare the environment

Hope that helps