GC duration concerning for jobtracker in CDH5.9 cluster

[#1]

Hi,

We are getting warning (concerning) messages for mapreduce1  jobtracker frequently on our Cloudera CDH 5.9 cluster. Currently java heap size of jobtracker is 4.11GB and it's 14-node cluster.

Below is the warning message that appears regarding the garbage collection.

Test of whether this role spends too much time in garbage collection.

Concerning:Average time spent in garbage collection was 20.6 seconds(34.28%)per minute over the previous  5 minute(s).Warning threshold:30.00%

Can someone please help me to get rid of the issue?

Thanks,

Priya

[#2 · Accepted solution]

@cdhhadoop

Try the below, it may help you

Cm -> Yarn -> Configuration -> "Java Heap Size of NodeManager in Bytes" Get the current value like 1GB or 2GB, etc... Increase one extra GB, ex: if it is 1GB, increase it to 2GB

(or) 

Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

restart yarn as needed

[#3]

@saranvisa,

Thanks for the reply . I will try the option you mentioned .

[#4]

Raising the heap should mostly help with this, as @saranvisa points out. The GC is usually driven by inadequate free heap memory, and adding more allocated heap memory helps with that. However, if the GCs are occurring without any change/increase in load, it could also be related to history retention configurations of the JobTracker.

> Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

I'd not recommend doing this, as it only hides the problem - You should be concerned if there's excessive GC due to heap pressure, and the default alert window and thresholds covers it well. Only increase it if you're absolutely sure the GC spikes are temporary and workload-driven but non-concerning, which doesn't seem determined yet.

Also, if you want to check the heap usage before increasing the heap by any value, you may see it in the heap charts shown on the JobTracker instance page of CM.

[#5]

@HarshJ
Thanks for inputs. I checked heap charts on jobtracker instance and it's hitting the maximum heap value frequently and then reducing to little less value.

Also, there has not been any change/increase in load.
I checked the jobtracker logs but couldn't find any pauses as logging is not enabled for GC.

Can you please let me know what are the history retention configurations of jobtracker?

Can you please suggest me how to identify the reason behind the GC taking significant time?

Thanks,
Priya