question Re: GC duration concerning for jobtracker in CDH5.9 cluster in Archives of Support Questions (Read Only)

GC duration concerning for jobtracker in CDH5.9 cluster

cdhhadoop — Mon, 13 Nov 2017 13:42:31 GMT

Hi,

We are getting warning (concerning) messages for mapreduce1 jobtracker frequently on our Cloudera CDH 5.9 cluster. Currently java heap size of jobtracker is 4.11GB and it's 14-node cluster.

Below is the warning message that appears regarding the garbage collection.

Test of whether this role spends too much time in garbage collection.

Concerning:Average time spent in garbage collection was 20.6 seconds(34.28%)per minute over the previous 5 minute(s).Warning threshold:30.00%

Can someone please help me to get rid of the issue?

Thanks,

Priya

Re: GC duration concerning for jobtracker in CDH5.9 cluster

saranvisa — Mon, 13 Nov 2017 15:58:23 GMT

@cdhhadoop

Try the below, it may help you

Cm -> Yarn -> Configuration -> "Java Heap Size of NodeManager in Bytes" Get the current value like 1GB or 2GB, etc... Increase one extra GB, ex: if it is 1GB, increase it to 2GB

(or)

Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

restart yarn as needed

Re: GC duration concerning for jobtracker in CDH5.9 cluster

cdhhadoop — Tue, 14 Nov 2017 04:26:59 GMT

@saranvisa,

Thanks for the reply . I will try the option you mentioned .

Re: GC duration concerning for jobtracker in CDH5.9 cluster

Harsh J — Tue, 14 Nov 2017 08:40:02 GMT

Raising the heap should mostly help with this, as @saranvisa points out. The GC is usually driven by inadequate free heap memory, and adding more allocated heap memory helps with that. However, if the GCs are occurring without any change/increase in load, it could also be related to history retention configurations of the JobTracker.

> Cm -> Yarn -> Configuration -> "Garbage Collection Duration Monitoring Period" Increase it from 5 mins to 10 mins

I'd not recommend doing this, as it only hides the problem - You should be concerned if there's excessive GC due to heap pressure, and the default alert window and thresholds covers it well. Only increase it if you're absolutely sure the GC spikes are temporary and workload-driven but non-concerning, which doesn't seem determined yet.

Also, if you want to check the heap usage before increasing the heap by any value, you may see it in the heap charts shown on the JobTracker instance page of CM.

Re: GC duration concerning for jobtracker in CDH5.9 cluster

cdhhadoop — Tue, 14 Nov 2017 09:41:37 GMT

@HarshJ
Thanks for inputs. I checked heap charts on jobtracker instance and it's hitting the maximum heap value frequently and then reducing to little less value.

Also, there has not been any change/increase in load.
I checked the jobtracker logs but couldn't find any pauses as logging is not enabled for GC.

Can you please let me know what are the history retention configurations of jobtracker?

Can you please suggest me how to identify the reason behind the GC taking significant time?

Thanks,
Priya