Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

GC Duration warnings on CDH 5.5.1 cluster

Solved Go to solution

GC Duration warnings on CDH 5.5.1 cluster

Contributor

Hi,

 

We recently started getting the GC warnings on our Yarn and all nodes of Production cluster frequently, so want to avoid this warnings. Please someone can help us to resolve this issue as soon as possible.

 

Following is the warning:

 

Test of whether this role spends too much time in garbage collection. Concerning

Average time spent in garbage collection was 19.8 second(s) (32.97%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.
1 ACCEPTED SOLUTION

Accepted Solutions

Re: GC Duration warnings on CDH 5.5.1 cluster

Rising Star

you will need to append "

 -Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

"

to the property  Java Configuration Options for Node Manager, and then restart Node Managers. You'll want to change  the numbers for -Xmx and -XX:MaxPermSize to values bigger than what you currently have for Node Manager. Please try to do this in a test enviroment first as this may impact on your production cluster.

13 REPLIES 13

Re: GC Duration warnings on CDH 5.5.1 cluster

Rising Star

What is exactly the role that had this warning message?

Re: GC Duration warnings on CDH 5.5.1 cluster

Contributor

Node Manager

Re: GC Duration warnings on CDH 5.5.1 cluster

Contributor

gc.png

Highlighted

Re: GC Duration warnings on CDH 5.5.1 cluster

Contributor

Following is the default configuration for garbage collection:

 

Screen Shot 2016-05-02 at 2.40.32 PM.png

Re: GC Duration warnings on CDH 5.5.1 cluster

Rising Star

Jais, for some reason, the configuration is not visiable to me. Can you copy and paste it here as text?

Re: GC Duration warnings on CDH 5.5.1 cluster

Contributor
I/O Sort Factor
mapreduce.task.io.sort.factor    ------> 64
 
Client Java Configuration Options -------- > -Djava.net.preferIPv4Stack=true
 
Java Configuration Options for JobHistory Server ------> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 
Java Configuration Options for NodeManager ------>  -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 
Java Configuration Options for ResourceManager -----> -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled
 
Garbage Collection Duration Thresholds--------->  Warning: 30.0 Critical: 60.0
Garbage Collection Duration Monitoring Period ------ > 5 minute(s)

Re: GC Duration warnings on CDH 5.5.1 cluster

Rising Star

Which version of java are you using? I do not know much about GC tuning, but heap size can lead to GC pause sometimes, you can try to increaes the heap size of Node Manager if that is possible. You can also try to collect gc logs by adding java options temporarily "-XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:PrintFLSStatistics=2"

Re: GC Duration warnings on CDH 5.5.1 cluster

Contributor

Java version we are using is "1.8.0_51".

 

What property do we need to reset for increasing heap size.

 

Is it 

 

Client Java Heap Size 
 
Gateway Default Group  -------------> 825955249 B = 787.69 MiB
 
If yes, by how much should we increase the size,
 
Its a 13 Node cluster

 

Re: GC Duration warnings on CDH 5.5.1 cluster

Rising Star

you will need to append "

 -Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

"

to the property  Java Configuration Options for Node Manager, and then restart Node Managers. You'll want to change  the numbers for -Xmx and -XX:MaxPermSize to values bigger than what you currently have for Node Manager. Please try to do this in a test enviroment first as this may impact on your production cluster.