HDFS Storage Capacity Usage Alert

[#1]

Hi all,

My ambari server has this alert.

HDFS Storage Capacity Usage (Daily)

I find this alert in this doc but no information for handle it.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-user-guide/content/hdfs_service_a...

This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a day period.

The variance for this alert is 2,015,674,228B which is 73% of the 2,750,867,892B average (1,375,433,946B is the limit)

Anyone can help to resolve this issue.?

[#2]

@Hoang Le

You can set the alert threshold based on your requirement.

Ambari IU --> Alerts (Tab) --> "Alert Definition Filter" Search for "HDFS Storage Capacity Usage" --> Click on "Edit" button

Now you can define a new "Growth Rate" CRITICAL & WARNING thresholds.

Also if your HDFS has some unwanted data in the ".trash" then you should clean it. Please see: https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#expunge

Also you should check your HDFS filesystem usage to validate the alert is correct or False.

# su - hdfs

# hdfs dfs -df -h
Filesystem          Size   Used  Available  Use%
hdfs://My-NN-HA  310.5 G  8.9 G    200.0 G    3%

.

[#3]

@jss

I flow you to check HDFS Storage Capacity Usage in Alert tab.

and it have Default setting:

so in my first post

is that the number 2,015,674,228B is Byte used by HDFS average daily?

and if true so how i change limit 1,375,433,946B ?

[#4]

@Hoang Le have you been able to overcome this issue?

I am having the same alerts and the alert is always there.

I haven't started using HDFS, haven't put and files either. I would like to know the reason if you found out. Also I formatted hdfs manually. Before this format, alerts were not critical.

[#5]

@Sedat Kestepe

you can increase growth rate and capacity to avoid this alert.

[#6]

Formatting hdfs is a critical task and requires the correct approach. In my case I formatted namenodes as journal nodes down. This led to consistency problems of journal quorum, nn1 and nn2. txids, editlog were different. For this reason alert was critical.

[#7]

I have changed the Growth rate but still I am getting the warning. can anyone help me storage-growth-rate.png

after the changes after-growth-rate-changes.png

,

[#8]

Maybe it is reasonable to get this alert when starting a new cluster, but annoying if you are just testing how is HDP.

One possible solution is to increase the "Check Count", or to just disable the alert.