Created 05-04-2017 05:02 PM
Running HDP 2.5.3 and ambari 2.4.2
I am having trouble getting some of the metrics and graphs to display.
Dashboard page displays graphs OK
Service page displays graphs OK except for Ambari Metrics - graphs just show spinning circles
Main hosts pages do not show Disk Usage or Load Avg.
Specific host page sometimes (rarely) shows graphs, usually shows spinners. But if I go back to the main hosts page, disk usage and load avg will display briefly for that host, then disappear.
I have seen posts regarding the restart of ntp servers - time on all hosts are synced together. I have seen a post regarding setting up a proxy host but the cluster is on an isolated network with a local repository - no internet access.
Any ideas?
Created 05-04-2017 06:47 PM
Please check if you could find any errors in Ambari metrics collector log. You could try clearing your hbase root/tmp directories if this would have caused by meta data corruption. Please note that if you remove hbase.rootdir, you could loose your historic AMS data. Please check the below URL.
https://community.hortonworks.com/questions/73577/problem-with-ambari-metrics.html
Ambari metrics monitors are used for collecting Host Metrics and sent to Ambari metrics collector. Service Metrics would be directly sent to the collector.
Created 05-04-2017 07:16 PM
@Jon Page How big is your cluster? It could be that AMS is not properly configured per cluster size or AMS HBase is writing to a contended disk. As a quick solution you could try whitelisting AMS data which reduces the write load and will give perf improvement.
https://issues.apache.org/jira/browse/AMBARI-16969
Configs: (make sure the whitelist file exists (came with the rpm))
ams-site :: timeline.metrics.whitelist.file = /etc/ambari-metrics-collector/conf/metrics_whitelist
ams-site : timeline.metrics.precision.table.durability = ASYNC_WAL
ams-site : timeline.metrics.aggregate.tables.durability = ASYNC_WAL
Created 05-05-2017 07:19 PM
43 nodes in total
I do not have the metrics_whitelist file in /etc/ambari-metrics-collector/conf
Created 05-05-2017 05:53 AM
Could you try this
Logon the Ambari server clear the contents of
# /var/lib/ambari-metrics- collector/hbase-tmp/zookeeper/*
Then restart the AMS that should help
Created 05-05-2017 02:40 PM
I think my problem is with a FW. If I stop the FW on and ambari server and the ambari-metrics-collector server, things seem OK. Besides 6188, are there any other ports that AMC use?
Created 05-05-2017 06:49 PM
If you are using AMS in embedded mode ZK quorum port would be 61181. But in distributed mode it is 2181 as it would be using cluster Zookeeper quorum. You might need to consider this port as well.
Created 05-05-2017 07:18 PM
running embedded mode. Added the ports, restarted iptables and ambari-metrics. Still getting spinners for graphs on the ambari-metrics page and on individual host pages, everything else seems to be OK. I haven't tried blowing away the var/lib/ambari-metrics directory
No errors in any of the logs
Created 05-05-2017 09:42 PM
The metric monitors send data to collector on 6188 port.
Did you check the log on the metric monitors? /var/log/ambari-metrics-monitor/ambari-metrics-monitor.out
HMaster port UI port is 61310.
Created 05-06-2017 08:45 AM
Which firewall are you talking of ? One of the pre-requisites is diabling the FW/TP etc did you execute those steps? Check tht these steps were all successful Prepare the environment
Hope that helps