Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Super Guru

This has been observed for Ambari 1.7. I know Ambari 1.7 is very old now however if some people are still using it and facing same issue then this post can save your lot of time!

.

We know that Ambari 1.7 has Ganglia for reporting metrics.

.

Our issue was, we were unable to get service metrics for YARN service, below was the exact scenario:

1. In Ganglia UI, I was able to get graphs for the YARN Metrics.

2. In Ambari UI, I was able to see metrics for other services like HDFS etc.

3. This issue was observed on one of our customer's cluster.

4. I did setup same cluster on my local system however I was not able to reproduce this issue.

5. Only difference in customer's and my cluster was, customer was having RM HA and I had installed single node instance.

6. There was no error while getting metrics, please see below screenshot when metrics were not visible.

10215-yarn.png

.

How to troubleshoot:

1. Click on any of the metrics, say 'Nodemanagers' --> This will open graph in big window.

2. Open the developer tools from Chrome/Firefox and inspect the network activities.

3. Notice the REST call from where its trying to fetch metrics.

4. Now do the RM failover and try to follow step 1-3 again.

5. Same REST call? No difference.

6. If you flip the RMs, Graphs will start populating the Data.

.

Root Cause:

If you look at hadoop-metrics2.properties file, it has only one RM host(initial rm1) hardcoded for resourcemanager.sink.ganglia.servers

resourcemanager.sink.ganglia.servers=rm1.crazyadmins.com:8664

.

Workaround:

Make same RM host as active RM which you get in REST API as a output of troubleshooting step 4.

.

Permanent fix:

Edit /etc/hadoop/conf/hadoop-metrics2.properties file and add another RM host:

E.g.

resourcemanager.sink.ganglia.servers=rm1.crazyadmins.com:8664,rm2.crazyadmins.com:8664

Note - This file is not managed by Ambari 1.7. Feel free to modify on both the RM hosts and restart RMs via Ambari after the modifications.

.

Hope you enjoyed this article! Please comment if you have any questions. Happy Hadooping!!

351 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 07:32 AM
Updated by:
 
Contributors
Top Kudoed Authors