Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

GC Duration warnings on CDH 5.5.1 cluster

avatar
Expert Contributor

Hi,

 

We recently started getting the GC warnings on our Yarn and all nodes of Production cluster frequently, so want to avoid this warnings. Please someone can help us to resolve this issue as soon as possible.

 

Following is the warning:

 

Test of whether this role spends too much time in garbage collection. Concerning

Average time spent in garbage collection was 19.8 second(s) (32.97%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.
1 ACCEPTED SOLUTION

avatar
Rising Star

you will need to append "

 -Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

"

to the property  Java Configuration Options for Node Manager, and then restart Node Managers. You'll want to change  the numbers for -Xmx and -XX:MaxPermSize to values bigger than what you currently have for Node Manager. Please try to do this in a test enviroment first as this may impact on your production cluster.

View solution in original post

13 REPLIES 13

avatar
Rising Star

What is exactly the role that had this warning message?

avatar
Expert Contributor

Node Manager

avatar
Expert Contributor

gc.png

avatar
Expert Contributor

Following is the default configuration for garbage collection:

 

Screen Shot 2016-05-02 at 2.40.32 PM.png

avatar
Rising Star

Jais, for some reason, the configuration is not visiable to me. Can you copy and paste it here as text?

avatar
Expert Contributor
I/O Sort Factor
mapreduce.task.io.sort.factor    ------> 64
 
Client Java Configuration Options -------- > -Djava.net.preferIPv4Stack=true
 
Java Configuration Options for JobHistory Server ------> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 
Java Configuration Options for NodeManager ------>  -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 
Java Configuration Options for ResourceManager -----> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 
Garbage Collection Duration Thresholds--------->  Warning: 30.0 Critical: 60.0
Garbage Collection Duration Monitoring Period ------ > 5 minute(s)

avatar
Rising Star

Which version of java are you using? I do not know much about GC tuning, but heap size can lead to GC pause sometimes, you can try to increaes the heap size of Node Manager if that is possible. You can also try to collect gc logs by adding java options temporarily "-XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:PrintFLSStatistics=2"

avatar
Expert Contributor

Java version we are using is "1.8.0_51".

 

What property do we need to reset for increasing heap size.

 

Is it 

 

Client Java Heap Size 
 
Gateway Default Group  -------------> 825955249 B = 787.69 MiB
 
If yes, by how much should we increase the size,
 
Its a 13 Node cluster

 

avatar
Rising Star

you will need to append "

 -Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

"

to the property  Java Configuration Options for Node Manager, and then restart Node Managers. You'll want to change  the numbers for -Xmx and -XX:MaxPermSize to values bigger than what you currently have for Node Manager. Please try to do this in a test enviroment first as this may impact on your production cluster.