Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

GC duration concerning for jobtracker in CDH5.9 cluster

avatar
Rising Star

Hi,

 

We are getting warning (concerning) messages for mapreduce1  jobtracker frequently on our Cloudera CDH 5.9 cluster. Currently java heap size of jobtracker is 4.11GB and it's 14-node cluster.

 


Below is the warning message that appears regarding the garbage collection.

 

Test of whether this role spends too much time in garbage collection.

 

Concerning:Average time spent in garbage collection was 20.6 seconds(34.28%)per minute over the previous  5 minute(s).Warning threshold:30.00%

 

 
Can someone please help me to get rid of the issue?
 
Thanks,
Priya
1 ACCEPTED SOLUTION

avatar
Champion

@cdhhadoop

 

Try the below, it may help you

 

Cm -> Yarn -> Configuration -> "Java Heap Size of NodeManager in Bytes" Get the current value like 1GB or 2GB, etc... Increase one extra GB, ex: if it is 1GB, increase it to 2GB

 

(or) 

 

Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

 

restart yarn as needed

View solution in original post

4 REPLIES 4

avatar
Champion

@cdhhadoop

 

Try the below, it may help you

 

Cm -> Yarn -> Configuration -> "Java Heap Size of NodeManager in Bytes" Get the current value like 1GB or 2GB, etc... Increase one extra GB, ex: if it is 1GB, increase it to 2GB

 

(or) 

 

Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

 

restart yarn as needed

avatar
Rising Star
@saranvisa,

Thanks for the reply . I will try the option you mentioned .

avatar
Mentor
Raising the heap should mostly help with this, as @saranvisa points out. The GC is usually driven by inadequate free heap memory, and adding more allocated heap memory helps with that. However, if the GCs are occurring without any change/increase in load, it could also be related to history retention configurations of the JobTracker.

> Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

I'd not recommend doing this, as it only hides the problem - You should be concerned if there's excessive GC due to heap pressure, and the default alert window and thresholds covers it well. Only increase it if you're absolutely sure the GC spikes are temporary and workload-driven but non-concerning, which doesn't seem determined yet.

Also, if you want to check the heap usage before increasing the heap by any value, you may see it in the heap charts shown on the JobTracker instance page of CM.

avatar
Rising Star
@HarshJ
Thanks for inputs. I checked heap charts on jobtracker instance and it's hitting the maximum heap value frequently and then reducing to little less value.

Also, there has not been any change/increase in load.
I checked the jobtracker logs but couldn't find any pauses as logging is not enabled for GC.

Can you please let me know what are the history retention configurations of jobtracker?

Can you please suggest me how to identify the reason behind the GC taking significant time?

Thanks,
Priya