Community Articles
Find and share helpful community-sourced technical articles
Labels (1)
Super Guru

This has been observed for Ambari 1.7. I know Ambari 1.7 is very old now however if some people are still using it and facing same issue then this post can save your lot of time!

.

We know that Ambari 1.7 has Ganglia for reporting metrics.

.

Our issue was, we were unable to get service metrics for YARN service, below was the exact scenario:

1. In Ganglia UI, I was able to get graphs for the YARN Metrics.

2. In Ambari UI, I was able to see metrics for other services like HDFS etc.

3. This issue was observed on one of our customer's cluster.

4. I did setup same cluster on my local system however I was not able to reproduce this issue.

5. Only difference in customer's and my cluster was, customer was having RM HA and I had installed single node instance.

6. There was no error while getting metrics, please see below screenshot when metrics were not visible.

10215-yarn.png

.

How to troubleshoot:

1. Click on any of the metrics, say 'Nodemanagers' --> This will open graph in big window.

2. Open the developer tools from Chrome/Firefox and inspect the network activities.

3. Notice the REST call from where its trying to fetch metrics.

4. Now do the RM failover and try to follow step 1-3 again.

5. Same REST call? No difference.

6. If you flip the RMs, Graphs will start populating the Data.

.

Root Cause:

If you look at hadoop-metrics2.properties file, it has only one RM host(initial rm1) hardcoded for resourcemanager.sink.ganglia.servers

resourcemanager.sink.ganglia.servers=rm1.crazyadmins.com:8664

.

Workaround:

Make same RM host as active RM which you get in REST API as a output of troubleshooting step 4.

.

Permanent fix:

Edit /etc/hadoop/conf/hadoop-metrics2.properties file and add another RM host:

E.g.

resourcemanager.sink.ganglia.servers=rm1.crazyadmins.com:8664,rm2.crazyadmins.com:8664

Note - This file is not managed by Ambari 1.7. Feel free to modify on both the RM hosts and restart RMs via Ambari after the modifications.

.

Hope you enjoyed this article! Please comment if you have any questions. Happy Hadooping!!

644 Views