Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Data Node (DN) Heap Usage Alerts in Ambari

Solved Go to solution

Data Node (DN) Heap Usage Alerts in Ambari

New Contributor

There are lot of queries which I have read w.r.t Data Node Heap Usage Alerts in Ambari. Consider the screenshot below:

13014-heap.png

We have 3 datanode, 1 name node and 1 utility node in our cluster. The 3 rows above heap usage of our 3 data node(DN).

On one DN we see that heap usage has gone to 91%. What I observerd is, once it reached 91% , I believe GC happens and heap usage reduces to 10% suddenly.

This behaviour repeats, after some time DN which has 50% heap usage will reach up to 91% and GC happens and heap usage reduces to 10%...and then same thing happens with DN which has 34% heap usage.

This is happenning in a circular fashion.

I needed help as to where can I change the setting, so that GC happens not at around 91% but at around 70%.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Data Node (DN) Heap Usage Alerts in Ambari

Hi @Akash S, this is a known issue and it is benign. You can safely ignore the alerts. If you want to avoid seeing them you can append the following options to HADOOP_DATANODE_OPTS via Ambari and restart DataNodes.

-XX:CMSInitiatingOccupancyFraction=70-XX:+UseCMSInitiatingOccupancyOnly-XX:ConcGCThreads=8-XX:+UseConcMarkSweepGC

Also asked here:

https://community.hortonworks.com/questions/85961/ambari-alert-datanode-heapsize-alert.html#answer-8...

3 REPLIES 3

Re: Data Node (DN) Heap Usage Alerts in Ambari

Super Guru
@Akash S

Are you seeing any performance issues with this? If not, then this is just your Garbage collection working the way it does. In fact it may not report how old generation objects are collection correctly. Your heap is also large (normal for hadoop -not for Java), so after a while it runs "stop the world" garbage collection and you see heap usage going down.

This is likely not an issue but just to dig deeper, which Garbage collector are you using?

Highlighted

Re: Data Node (DN) Heap Usage Alerts in Ambari

Hi @Akash S, this is a known issue and it is benign. You can safely ignore the alerts. If you want to avoid seeing them you can append the following options to HADOOP_DATANODE_OPTS via Ambari and restart DataNodes.

-XX:CMSInitiatingOccupancyFraction=70-XX:+UseCMSInitiatingOccupancyOnly-XX:ConcGCThreads=8-XX:+UseConcMarkSweepGC

Also asked here:

https://community.hortonworks.com/questions/85961/ambari-alert-datanode-heapsize-alert.html#answer-8...

Re: Data Node (DN) Heap Usage Alerts in Ambari

New Contributor

This worked.

Don't have an account?
Coming from Hortonworks? Activate your account here