question Re: GC Duration warnings on CDH 5.5.1 cluster in Support Questions

GC Duration warnings on CDH 5.5.1 cluster

Jais — Fri, 16 Sep 2022 10:16:40 GMT

Hi,

We recently started getting the GC warnings on our Yarn and all nodes of Production cluster frequently, so want to avoid this warnings. Please someone can help us to resolve this issue as soon as possible.

Following is the warning:

Test of whether this role spends too much time in garbage collection. Concerning

Average time spent in garbage collection was 19.8 second(s) (32.97%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.

Re: GC Duration warnings on CDH 5.5.1 cluster

haibochen — Mon, 02 May 2016 21:16:07 GMT

What is exactly the role that had this warning message?

Re: GC Duration warnings on CDH 5.5.1 cluster

Jais — Mon, 02 May 2016 21:26:04 GMT

Node Manager

Re: GC Duration warnings on CDH 5.5.1 cluster

Jais — Mon, 02 May 2016 21:28:22 GMT

Re: GC Duration warnings on CDH 5.5.1 cluster

Jais — Mon, 02 May 2016 21:42:12 GMT

Following is the default configuration for garbage collection:

Re: GC Duration warnings on CDH 5.5.1 cluster

haibochen — Mon, 02 May 2016 22:12:25 GMT

Jais, for some reason, the configuration is not visiable to me. Can you copy and paste it here as text?

Re: GC Duration warnings on CDH 5.5.1 cluster

Jais — Mon, 02 May 2016 22:19:31 GMT

I/O Sort Factor

mapreduce.task.io.sort.factor ------> 64

Client Java Configuration Options -------- > -Djava.net.preferIPv4Stack=true

Java Configuration Options for JobHistory Server ------> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled

Java Configuration Options for NodeManager ------> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled

Java Configuration Options for ResourceManager -----> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled

Garbage Collection Duration Thresholds---------> Warning: 30.0 Critical: 60.0

Garbage Collection Duration Monitoring Period ------ > 5 minute(s)

Re: GC Duration warnings on CDH 5.5.1 cluster

haibochen — Mon, 02 May 2016 23:19:54 GMT

Which version of java are you using? I do not know much about GC tuning, but heap size can lead to GC pause sometimes, you can try to increaes the heap size of Node Manager if that is possible. You can also try to collect gc logs by adding java options temporarily "-XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:PrintFLSStatistics=2"

Re: GC Duration warnings on CDH 5.5.1 cluster

Jais — Mon, 02 May 2016 23:44:53 GMT

Java version we are using is "1.8.0_51".

What property do we need to reset for increasing heap size.

Is it

Client Java Heap Size

Gateway Default Group -------------> 825955249 B = 787.69 MiB

If yes, by how much should we increase the size,

Its a 13 Node cluster

Re: GC Duration warnings on CDH 5.5.1 cluster

haibochen — Tue, 03 May 2016 00:24:11 GMT

you will need to append "

 -Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

to the property Java Configuration Options for Node Manager, and then restart Node Managers. You'll want to change the numbers for -Xmx and -XX:MaxPermSize to values bigger than what you currently have for Node Manager. Please try to do this in a test enviroment first as this may impact on your production cluster.

Re: GC Duration warnings on CDH 5.5.1 cluster

Jais — Tue, 03 May 2016 00:50:46 GMT

Can you please be more specific to this, like how to find out the current setting for following:

-Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

Re: GC Duration warnings on CDH 5.5.1 cluster

haibochen — Tue, 03 May 2016 01:04:40 GMT

http://stackoverflow.com/questions/9257361/how-to-check-heap-size-allocated-for-jvm-by-linux

Re: GC Duration warnings on CDH 5.5.1 cluster

Jais — Wed, 04 May 2016 17:48:34 GMT

Ok we applied the following configuration as recommended on our Dev Cluster and are seeing no issues.

property: Java Configuration Options for NodeManager:

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Xms512m -Xmx2048m -XX:PermSize=216m -XX:MaxPermSize=512m

But the point to note is GC Duration warnings had never happened on this cluster, it happened on our Production cluster.

So shall we go ahead and make changes on Production cluster or there are any other things we have to consider ?

Please any suggestions

Re: GC Duration warnings on CDH 5.5.1 cluster

haibochen — Thu, 05 May 2016 02:23:40 GMT

Is it possible for you to try the same settings in your production cluster? If your dev cluster is very close to production server in terms of workload, then I'd guess this will alleviate gc pause in your production cluster as well.