question Re: Help after moving Ambari metrics server in Support Questions

Help after moving Ambari metrics server

zack_riesland — Sat, 27 Feb 2016 03:03:57 GMT

I moved our ambari metrics server from a data node to an edge node.

I followed all of the directions outlined here:

http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.0.0/bk_ambari_reference_guide/content/ch_moving_the_ambari_metrics_collector.html

Now, ambari metrics kind of 'works', but not completely.

The main Ambari dashboard looks good - all of the metrics are populated.

But if I drill into HDFS, YARN, HBase, etc - only some of the metrics are there.

Many say "No data available".

I tried re-starting ambari server, but no luck.

I tried ctrl+shift+r to force a clean refresh of the GUI, and that didn't help either.

I know that all the nodes are pointing to the right place in /etc/ambari-metrics-monitor/conf/metric_monitor.ini

Where can I look to figure this out?

Re: Help after moving Ambari metrics server

aervits — Sat, 27 Feb 2016 03:12:43 GMT

@Zack Riesland let's do a few things. Do you have firewall rules per port? Is there a rule you need to add to your firewall for each monitor? What is the version of Ambari you're running? There were issues in Ambari metrics 2.1.x https://cwiki.apache.org/confluence/display/AMBARI/Known+Issues

Re: Help after moving Ambari metrics server

zack_riesland — Sat, 27 Feb 2016 03:31:15 GMT

We are on ambari 2.1.1

We run HA, w/o Kerberos or any security.

I restarted ambari server and then restarted all the monitors and it didn't help.

I don't know about firewalls on our server. Where would I look for this?

Thanks!

Re: Help after moving Ambari metrics server

aervits — Sat, 27 Feb 2016 03:33:21 GMT

I'll find out the exact firewall conf file in a bit but also look at our docs for DNS and proxy settings requirements, I've seen similar symptoms from other HCC entries. I'm not by computer so can't be of more use right now. Look at HCC for similar threads.

Re: Help after moving Ambari metrics server

aervits — Sat, 27 Feb 2016 03:38:51 GMT

heres one solution https://community.hortonworks.com/questions/9903/ambari-metrics-display-issue.html

Re: Help after moving Ambari metrics server

Ravi_G — Sat, 27 Feb 2016 03:40:25 GMT

@Zack Riesland

After moving AMS, you need to restart the HDFS, YARN, HBase services so that they can start sending data to new AMS host.

Re: Help after moving Ambari metrics server

aervits — Sat, 27 Feb 2016 03:40:36 GMT

And this https://community.hortonworks.com/questions/4726/ambari-metric-collector-error-sending-metric-to-se.html without logs its hard to debug

Re: Help after moving Ambari metrics server

zack_riesland — Sat, 27 Feb 2016 03:47:30 GMT

Thanks @rgangappa, that seems to be what I needed. @Artem Ervits, that original instructions document probably needs this.

Thanks!

Re: Help after moving Ambari metrics server

aervits — Sat, 27 Feb 2016 03:49:09 GMT

@Zack Riesland you're right, seems like an obvious thing to restart the services, usually it will have an orange circle next to all services needing a restart. Good catch @rgangappa

Re: Help after moving Ambari metrics server

zack_riesland — Sat, 27 Feb 2016 03:51:32 GMT

No orange circle. No indication to restart services. FYI.

Re: Help after moving Ambari metrics server

aervits — Sat, 27 Feb 2016 03:54:03 GMT

@Zack Riesland I tagged to notify docs to add this. Thanks for opening the thread!