Running HDP 2.5.3 and ambari 2.4.2
I am having trouble getting some of the metrics and graphs to display.
Dashboard page displays graphs OK
Service page displays graphs OK except for Ambari Metrics - graphs just show spinning circles
Main hosts pages do not show Disk Usage or Load Avg.
Specific host page sometimes (rarely) shows graphs, usually shows spinners. But if I go back to the main hosts page, disk usage and load avg will display briefly for that host, then disappear.
I have seen posts regarding the restart of ntp servers - time on all hosts are synced together. I have seen a post regarding setting up a proxy host but the cluster is on an isolated network with a local repository - no internet access.
Please check if you could find any errors in Ambari metrics collector log. You could try clearing your hbase root/tmp directories if this would have caused by meta data corruption. Please note that if you remove hbase.rootdir, you could loose your historic AMS data. Please check the below URL.
Ambari metrics monitors are used for collecting Host Metrics and sent to Ambari metrics collector. Service Metrics would be directly sent to the collector.
@Jon Page How big is your cluster? It could be that AMS is not properly configured per cluster size or AMS HBase is writing to a contended disk. As a quick solution you could try whitelisting AMS data which reduces the write load and will give perf improvement.
Configs: (make sure the whitelist file exists (came with the rpm))
ams-site :: timeline.metrics.whitelist.file = /etc/ambari-metrics-collector/conf/metrics_whitelist
ams-site : timeline.metrics.precision.table.durability = ASYNC_WAL
ams-site : timeline.metrics.aggregate.tables.durability = ASYNC_WAL
I think my problem is with a FW. If I stop the FW on and ambari server and the ambari-metrics-collector server, things seem OK. Besides 6188, are there any other ports that AMC use?
If you are using AMS in embedded mode ZK quorum port would be 61181. But in distributed mode it is 2181 as it would be using cluster Zookeeper quorum. You might need to consider this port as well.
running embedded mode. Added the ports, restarted iptables and ambari-metrics. Still getting spinners for graphs on the ambari-metrics page and on individual host pages, everything else seems to be OK. I haven't tried blowing away the var/lib/ambari-metrics directory
No errors in any of the logs
The metric monitors send data to collector on 6188 port.
Did you check the log on the metric monitors? /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out
HMaster port UI port is 61310.