Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HDFS Storage Capacity Usage Alert

HDFS Storage Capacity Usage Alert

New Contributor

Hi all,

My ambari server has this alert.

HDFS Storage Capacity Usage (Daily)

I find this alert in this doc but no information for handle it.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.0.1/bk_ambari-user-guide/content/hdfs_service_a...

This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a day period.

The variance for this alert is 2,015,674,228B which is 73% of the 2,750,867,892B average (1,375,433,946B is the limit)

Anyone can help to resolve this issue.?

7 REPLIES 7

Re: HDFS Storage Capacity Usage Alert

@Hoang Le

You can set the alert threshold based on your requirement.

Ambari IU --> Alerts (Tab) --> "Alert Definition Filter" Search for "HDFS Storage Capacity Usage" --> Click on "Edit" button

Now you can define a new "Growth Rate" CRITICAL & WARNING thresholds.

Also if your HDFS has some unwanted data in the ".trash" then you should clean it. Please see: https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#expunge

Also you should check your HDFS filesystem usage to validate the alert is correct or False.

# su - hdfs

# hdfs dfs -df -h
Filesystem          Size   Used  Available  Use%
hdfs://My-NN-HA  310.5 G  8.9 G    200.0 G    3%

.

Re: HDFS Storage Capacity Usage Alert

New Contributor

@jss

I flow you to check HDFS Storage Capacity Usage in Alert tab.

and it have Default setting:

9264-alert.png

so in my first post

is that the number 2,015,674,228B is Byte used by HDFS average daily?

and if true so how i change limit 1,375,433,946B ?

Re: HDFS Storage Capacity Usage Alert

Rising Star

@Hoang Le have you been able to overcome this issue?

I am having the same alerts and the alert is always there.

I haven't started using HDFS, haven't put and files either. I would like to know the reason if you found out. Also I formatted hdfs manually. Before this format, alerts were not critical.

Re: HDFS Storage Capacity Usage Alert

New Contributor
@Sedat Kestepe

you can increase growth rate and capacity to avoid this alert.

Re: HDFS Storage Capacity Usage Alert

Rising Star

Formatting hdfs is a critical task and requires the correct approach. In my case I formatted namenodes as journal nodes down. This led to consistency problems of journal quorum, nn1 and nn2. txids, editlog were different. For this reason alert was critical.

Re: HDFS Storage Capacity Usage Alert

New Contributor

I have changed the Growth rate but still I am getting the warning. can anyone help me storage-growth-rate.png

after the changes after-growth-rate-changes.png

,


storage-growth-rate.pngafter-growth-rate-changes.png

Re: HDFS Storage Capacity Usage Alert

New Contributor

Maybe it is reasonable to get this alert when starting a new cluster, but annoying if you are just testing how is HDP.

One possible solution is to increase the "Check Count", or to just disable the alert.

68420-seleccion-075.png