Data Node (DN) Heap Usage Alerts in Ambari

[#1]

There are lot of queries which I have read w.r.t Data Node Heap Usage Alerts in Ambari. Consider the screenshot below:

We have 3 datanode, 1 name node and 1 utility node in our cluster. The 3 rows above heap usage of our 3 data node(DN).

On one DN we see that heap usage has gone to 91%. What I observerd is, once it reached 91% , I believe GC happens and heap usage reduces to 10% suddenly.

This behaviour repeats, after some time DN which has 50% heap usage will reach up to 91% and GC happens and heap usage reduces to 10%...and then same thing happens with DN which has 34% heap usage.

This is happenning in a circular fashion.

I needed help as to where can I change the setting, so that GC happens not at around 91% but at around 70%.

[#2]

@Akash S

Are you seeing any performance issues with this? If not, then this is just your Garbage collection working the way it does. In fact it may not report how old generation objects are collection correctly. Your heap is also large (normal for hadoop -not for Java), so after a while it runs "stop the world" garbage collection and you see heap usage going down.

This is likely not an issue but just to dig deeper, which Garbage collector are you using?

[#3 · Accepted solution]

Hi @Akash S, this is a known issue and it is benign. You can safely ignore the alerts. If you want to avoid seeing them you can append the following options to HADOOP_DATANODE_OPTS via Ambari and restart DataNodes.

-XX:CMSInitiatingOccupancyFraction=70-XX:+UseCMSInitiatingOccupancyOnly-XX:ConcGCThreads=8-XX:+UseConcMarkSweepGC

Also asked here:

https://community.hortonworks.com/questions/85961/ambari-alert-datanode-heapsize-alert.html#answer-8...

[#4]

This worked.