GC Duration warnings on CDH 5.5.1 cluster

[#1]

Hi,

We recently started getting the GC warnings on our Yarn and all nodes of Production cluster frequently, so want to avoid this warnings. Please someone can help us to resolve this issue as soon as possible.

Following is the warning:

Test of whether this role spends too much time in garbage collection. Concerning

Average time spent in garbage collection was 19.8 second(s) (32.97%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.

[#2]

What is exactly the role that had this warning message?

[#3]

Node Manager

[#4]

[#5]

Following is the default configuration for garbage collection:

[#6]

Jais, for some reason, the configuration is not visiable to me. Can you copy and paste it here as text?

[#7]

I/O Sort Factor

mapreduce.task.io.sort.factor    ------> 64

Client Java Configuration Options -------- > -Djava.net.preferIPv4Stack=true

Java Configuration Options for JobHistory Server ------> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled

Java Configuration Options for NodeManager ------>  -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled

Java Configuration Options for ResourceManager -----> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled

Garbage Collection Duration Thresholds--------->  Warning: 30.0 Critical: 60.0

Garbage Collection Duration Monitoring Period ------ > 5 minute(s)

[#8]

Which version of java are you using? I do not know much about GC tuning, but heap size can lead to GC pause sometimes, you can try to increaes the heap size of Node Manager if that is possible. You can also try to collect gc logs by adding java options temporarily "-XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:PrintFLSStatistics=2"

[#9]

Java version we are using is "1.8.0_51".

What property do we need to reset for increasing heap size.

Is it 

Client Java Heap Size 

Gateway Default Group  -------------> 825955249 B = 787.69 MiB

If yes, by how much should we increase the size,

Its a 13 Node cluster

[#10 · Accepted solution]

you will need to append "

 -Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

"

to the property  Java Configuration Options for Node Manager, and then restart Node Managers. You'll want to change  the numbers for -Xmx and -XX:MaxPermSize to values bigger than what you currently have for Node Manager. Please try to do this in a test enviroment first as this may impact on your production cluster.