Created 02-26-2016 07:03 PM
I moved our ambari metrics server from a data node to an edge node.
I followed all of the directions outlined here:
Now, ambari metrics kind of 'works', but not completely.
The main Ambari dashboard looks good - all of the metrics are populated.
But if I drill into HDFS, YARN, HBase, etc - only some of the metrics are there.
Many say "No data available".
I tried re-starting ambari server, but no luck.
I tried ctrl+shift+r to force a clean refresh of the GUI, and that didn't help either.
I know that all the nodes are pointing to the right place in /etc/ambari-metrics-monitor/conf/metric_monitor.ini
Where can I look to figure this out?
Created 02-26-2016 07:40 PM
After moving AMS, you need to restart the HDFS, YARN, HBase services so that they can start sending data to new AMS host.
Created 02-26-2016 07:12 PM
@Zack Riesland let's do a few things. Do you have firewall rules per port? Is there a rule you need to add to your firewall for each monitor? What is the version of Ambari you're running? There were issues in Ambari metrics 2.1.x https://cwiki.apache.org/confluence/display/AMBARI/Known+Issues
Created 02-26-2016 07:31 PM
We are on ambari 2.1.1
We run HA, w/o Kerberos or any security.
I restarted ambari server and then restarted all the monitors and it didn't help.
I don't know about firewalls on our server. Where would I look for this?
Thanks!
Created 02-26-2016 07:33 PM
I'll find out the exact firewall conf file in a bit but also look at our docs for DNS and proxy settings requirements, I've seen similar symptoms from other HCC entries. I'm not by computer so can't be of more use right now. Look at HCC for similar threads.
Created 02-26-2016 07:38 PM
Created 02-26-2016 07:40 PM
And this https://community.hortonworks.com/questions/4726/ambari-metric-collector-error-sending-metric-to-se.... without logs its hard to debug
Created 02-26-2016 07:40 PM
After moving AMS, you need to restart the HDFS, YARN, HBase services so that they can start sending data to new AMS host.
Created 02-26-2016 07:47 PM
Thanks @rgangappa, that seems to be what I needed. @Artem Ervits, that original instructions document probably needs this.
Thanks!
Created 02-26-2016 07:49 PM
@Zack Riesland you're right, seems like an obvious thing to restart the services, usually it will have an orange circle next to all services needing a restart. Good catch @rgangappa
Created 02-26-2016 07:51 PM
No orange circle. No indication to restart services. FYI.