GC Duration warnings on CDH 5.5.1 cluster

[#1]

Hi,

We recently started getting the GC warnings on our Yarn and all nodes of Production cluster frequently, so want to avoid this warnings. Please someone can help us to resolve this issue as soon as possible.

Following is the warning:

Test of whether this role spends too much time in garbage collection. Concerning

Average time spent in garbage collection was 19.8 second(s) (32.97%) per minute over the previous 5 minute(s). Warning threshold: 30.00%.

[#2]

Can you please be more specific to this, like how to find out the current setting for following:

-Xms512m -Xmx1024m -XX:PermSize=216m -XX:MaxPermSize=512m

[#3]

http://stackoverflow.com/questions/9257361/how-to-check-heap-size-allocated-for-jvm-by-linux

[#4]

Ok we applied the following configuration as recommended on our Dev Cluster and are seeing no issues.

property: Java Configuration Options for NodeManager:

-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Xms512m -Xmx2048m -XX:PermSize=216m -XX:MaxPermSize=512m

But the point to note is GC Duration warnings had never happened on this cluster, it happened on our Production cluster. 

So shall we go ahead and make changes on Production cluster or there are any other things we have to consider ?

Please any suggestions

[#5]

Is it possible for you to try the same settings in your production cluster? If your dev cluster is very close to production server in terms of workload, then I'd guess this will alleviate gc pause in your production cluster as well.