Support Questions

Find answers, ask questions, and share your expertise

we haven't started using HDFS but we get service-level alert is triggered if the increase in storage capacity usage - why?

avatar

we have ambari cluster

we haven't started using HDFS but we get service-level alert is triggered if the increase in storage capacity usage - why?

59424-capture.png

59424-capture.png

please advice why?

why capacity is grow in spite we not use HDFS?

and what is the solotion for this?


Michael-Bronson
1 ACCEPTED SOLUTION

avatar
Master Mentor

@Michael Bronson

Even if we are not directly using HDFS on our own, there are some background jobs runs like Ambari Service checks that do write data on HDFS.

.

Looks like you are looking at the "HDFS Storage Capacity Usage (Weekly)" alert. This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a week period.

The default alert definition says:
Show a WARNING (YELLOW) message when the HDFS usage grows more than 10% in a week
Show a CRITICAL (RED) message when the HDFS usage grows more than 20% in a week

56538-weekely-hdfs-alert.png


With this kind of alert you will see a message also at the bottom something like following which i got in my environment:

The variance for this alert is 895,395,035B which is 13% of the 6,861,760,545B average (686,176,054B is the limit)


This says that my HDFS usage growth was 13% compared to last week so based on the threshold value ambari shows it as a WARNING.


These threshold values can be set based on different environments (some environments may be using HDFS heavily and some may not be) so we can set the percentage based on our requirement.

.

So please check the Alert Message that will show how much growth in your HDFS usage happened which caused this alert. There is nothing to worry about usually. We need to just adjust the threashould of this alert based on our need.

.

View solution in original post

4 REPLIES 4

avatar
Master Mentor

@Michael Bronson

Even if we are not directly using HDFS on our own, there are some background jobs runs like Ambari Service checks that do write data on HDFS.

.

Looks like you are looking at the "HDFS Storage Capacity Usage (Weekly)" alert. This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a week period.

The default alert definition says:
Show a WARNING (YELLOW) message when the HDFS usage grows more than 10% in a week
Show a CRITICAL (RED) message when the HDFS usage grows more than 20% in a week

56538-weekely-hdfs-alert.png


With this kind of alert you will see a message also at the bottom something like following which i got in my environment:

The variance for this alert is 895,395,035B which is 13% of the 6,861,760,545B average (686,176,054B is the limit)


This says that my HDFS usage growth was 13% compared to last week so based on the threshold value ambari shows it as a WARNING.


These threshold values can be set based on different environments (some environments may be using HDFS heavily and some may not be) so we can set the percentage based on our requirement.

.

So please check the Alert Message that will show how much growth in your HDFS usage happened which caused this alert. There is nothing to worry about usually. We need to just adjust the threashould of this alert based on our need.

.

avatar

I checked and we got - The variance for this alert is 6,233,496,816B which is 74% of the 8,471,125,314B average (1,694,225,063B is the limit) , so as I understand we cant do nothing about this , only to update the thershold - am I right ?

Michael-Bronson

avatar
Master Mentor

@Michael Bronson

We need to worry only when we see drastic growth in the HDFS usage. But if we see not much growth and still if we see the warning message then it indicates that the WARNING percentage threshold is not set as per our requirement. The default value 10% may not be suitable for small test clusters i think.

.

In one old HCC thread i tried explaining about this alert with some examples, may be that will help in getting better understanding about this alert:

https://community.hortonworks.com/questions/140101/what-does-hdfs-storage-capacity-usage-weekly-aler...

avatar

ok so in order to avoid this alarm we can set "Growth Rate" from 20% to 40% , or to increase minimum capacity from 1000 to 5000 , what is the better option ?

Michael-Bronson