Reply
Explorer
Posts: 56
Registered: ‎08-07-2017

GC duration concerning for jobtracker in CDH5.9 cluster

[ Edited ]

Hi,

 

We are getting warning (concerning) messages for mapreduce1  jobtracker frequently on our Cloudera CDH 5.9 cluster. Currently java heap size of jobtracker is 4.11GB and it's 14-node cluster.

 


Below is the warning message that appears regarding the garbage collection.

 

Test of whether this role spends too much time in garbage collection.

 

Concerning:Average time spent in garbage collection was 20.6 seconds(34.28%)per minute over the previous  5 minute(s).Warning threshold:30.00%

 

 
Can someone please help me to get rid of the issue?
 
Thanks,
Priya
Highlighted
Posts: 388
Topics: 11
Kudos: 60
Solutions: 34
Registered: ‎09-02-2016

Re: GC duration concerning for jobtracker in CDH5.9 cluster

@cdhhadoop

 

Try the below, it may help you

 

Cm -> Yarn -> Configuration -> "Java Heap Size of NodeManager in Bytes" Get the current value like 1GB or 2GB, etc... Increase one extra GB, ex: if it is 1GB, increase it to 2GB

 

(or) 

 

Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

 

restart yarn as needed

Explorer
Posts: 56
Registered: ‎08-07-2017

Re: GC duration concerning for jobtracker in CDH5.9 cluster

@saranvisa,

Thanks for the reply . I will try the option you mentioned .
Posts: 1,565
Kudos: 287
Solutions: 239
Registered: ‎07-31-2013

Re: GC duration concerning for jobtracker in CDH5.9 cluster

Raising the heap should mostly help with this, as @saranvisa points out. The GC is usually driven by inadequate free heap memory, and adding more allocated heap memory helps with that. However, if the GCs are occurring without any change/increase in load, it could also be related to history retention configurations of the JobTracker.

> Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

I'd not recommend doing this, as it only hides the problem - You should be concerned if there's excessive GC due to heap pressure, and the default alert window and thresholds covers it well. Only increase it if you're absolutely sure the GC spikes are temporary and workload-driven but non-concerning, which doesn't seem determined yet.

Also, if you want to check the heap usage before increasing the heap by any value, you may see it in the heap charts shown on the JobTracker instance page of CM.
Backline Customer Operations Engineer
Explorer
Posts: 56
Registered: ‎08-07-2017

Re: GC duration concerning for jobtracker in CDH5.9 cluster

@HarshJ
Thanks for inputs. I checked heap charts on jobtracker instance and it's hitting the maximum heap value frequently and then reducing to little less value.

Also, there has not been any change/increase in load.
I checked the jobtracker logs but couldn't find any pauses as logging is not enabled for GC.

Can you please let me know what are the history retention configurations of jobtracker?

Can you please suggest me how to identify the reason behind the GC taking significant time?

Thanks,
Priya

Announcements