I got below alert, Seems NN heap size was occupied too much. how to resolve this issue
The variance for this alert is 1,143MB which is 27% of the 4,224MB average (845MB is the limit)
These two links contain most of the information that is needed for tuning the Namenode heap.
Alerts come from Ambari, so if you want to tune that instead of Namenode heap, you can do that in Ambari.
There are three parameters on which is alert is triggered :
Growth Rate for Critical ,Growth Rate for Warning and Minimum Heap
Minimum Heap : if the heap usage of namenode is less that configured value then it will not trigger any message.
Growth Rate for Warning : We give a value for this parameter in %. Which means that if the deviation in heap usage of namenode is above the % of "Average of Namenode Heap Usage" then it should trigger Warning alert.
For Ex : The variance for this alert is 1,143MB which is 27% of the 4,224MB average (845MB is the limit)
Growth Rate for Critical : 50%
Growth Rate for Warning : 20 %
Minimum Heap 100 MB
Namenode Average heap usage : 4224 MB
Variance : 1143 MB
This means that 1143MB is the deviation of heap usage which is less than 27% of 4,224MB which is Average heap usage for your cluster.
So 1,143MB is greater than 845MB, hence it will trigger an alert.
We cannot really tune the NN heap based on this alert as this shows only the daily/weekly variance. It could happen if there was too much activity on the cluster just for that day, also I have seen this coming when we restart namenodes.
Please analyze the NN log and the GC log to see if there are any really long GC pauses and consistent over heap utilization, If everything looks normal I would just reconfigure the thresholds for these alerts in ambari. Please check this link to see if the NN heap meet the recommendations.
You can configure the ambari alert thresholds as per your requirements by analyzing Average namenode heap usage on your cluster. You need to identify or you can set a benchmark for heap usage and configure this alerts as per it.