Created on 02-01-2018 08:23 AM - edited 08-17-2019 09:26 PM
we have ambari cluster
we haven't started using HDFS but we get service-level alert is triggered if the increase in storage capacity usage - why?
please advice why?
why capacity is grow in spite we not use HDFS?
and what is the solotion for this?
Created on 02-01-2018 08:43 AM - edited 08-17-2019 09:26 PM
Even if we are not directly using HDFS on our own, there are some background jobs runs like Ambari Service checks that do write data on HDFS.
.
Looks like you are looking at the "HDFS Storage Capacity Usage (Weekly)" alert. This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a week period.
The default alert definition says:
Show a WARNING (YELLOW) message when the HDFS usage grows more than 10% in a week
Show a CRITICAL (RED) message when the HDFS usage grows more than 20% in a week
With this kind of alert you will see a message also at the bottom something like following which i got in my environment:
The variance for this alert is 895,395,035B which is 13% of the 6,861,760,545B average (686,176,054B is the limit)
This says that my HDFS usage growth was 13% compared to last week so based on the threshold value ambari shows it as a WARNING.
These threshold values can be set based on different environments (some environments may be using HDFS heavily and some may not be) so we can set the percentage based on our requirement.
.
So please check the Alert Message that will show how much growth in your HDFS usage happened which caused this alert. There is nothing to worry about usually. We need to just adjust the threashould of this alert based on our need.
.
Created on 02-01-2018 08:43 AM - edited 08-17-2019 09:26 PM
Even if we are not directly using HDFS on our own, there are some background jobs runs like Ambari Service checks that do write data on HDFS.
.
Looks like you are looking at the "HDFS Storage Capacity Usage (Weekly)" alert. This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a week period.
The default alert definition says:
Show a WARNING (YELLOW) message when the HDFS usage grows more than 10% in a week
Show a CRITICAL (RED) message when the HDFS usage grows more than 20% in a week
With this kind of alert you will see a message also at the bottom something like following which i got in my environment:
The variance for this alert is 895,395,035B which is 13% of the 6,861,760,545B average (686,176,054B is the limit)
This says that my HDFS usage growth was 13% compared to last week so based on the threshold value ambari shows it as a WARNING.
These threshold values can be set based on different environments (some environments may be using HDFS heavily and some may not be) so we can set the percentage based on our requirement.
.
So please check the Alert Message that will show how much growth in your HDFS usage happened which caused this alert. There is nothing to worry about usually. We need to just adjust the threashould of this alert based on our need.
.
Created 02-01-2018 08:49 AM
I checked and we got - The variance for this alert is 6,233,496,816B which is 74% of the 8,471,125,314B average (1,694,225,063B is the limit) , so as I understand we cant do nothing about this , only to update the thershold - am I right ?
Created 02-01-2018 09:02 AM
We need to worry only when we see drastic growth in the HDFS usage. But if we see not much growth and still if we see the warning message then it indicates that the WARNING percentage threshold is not set as per our requirement. The default value 10% may not be suitable for small test clusters i think.
.
In one old HCC thread i tried explaining about this alert with some examples, may be that will help in getting better understanding about this alert:
Created 02-01-2018 09:17 AM
ok so in order to avoid this alarm we can set "Growth Rate" from 20% to 40% , or to increase minimum capacity from 1000 to 5000 , what is the better option ?