My ambari server has this alert.
HDFS Storage Capacity Usage (Daily)
I find this alert in this doc but no information for handle it.
This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a day period.
The variance for this alert is 2,015,674,228B which is 73% of the 2,750,867,892B average (1,375,433,946B is the limit)
Anyone can help to resolve this issue.?
You can set the alert threshold based on your requirement.
Ambari IU --> Alerts (Tab) --> "Alert Definition Filter" Search for "HDFS Storage Capacity Usage" --> Click on "Edit" button
Now you can define a new "Growth Rate" CRITICAL & WARNING thresholds.
Also if your HDFS has some unwanted data in the ".trash" then you should clean it. Please see: https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#expunge
Also you should check your HDFS filesystem usage to validate the alert is correct or False.
# su - hdfs # hdfs dfs -df -h Filesystem Size Used Available Use% hdfs://My-NN-HA 310.5 G 8.9 G 200.0 G 3%
I flow you to check HDFS Storage Capacity Usage in Alert tab.
and it have Default setting:
so in my first post
is that the number 2,015,674,228B is Byte used by HDFS average daily?
and if true so how i change limit 1,375,433,946B ?
@Hoang Le have you been able to overcome this issue?
I am having the same alerts and the alert is always there.
I haven't started using HDFS, haven't put and files either. I would like to know the reason if you found out. Also I formatted hdfs manually. Before this format, alerts were not critical.
Formatting hdfs is a critical task and requires the correct approach. In my case I formatted namenodes as journal nodes down. This led to consistency problems of journal quorum, nn1 and nn2. txids, editlog were different. For this reason alert was critical.
Maybe it is reasonable to get this alert when starting a new cluster, but annoying if you are just testing how is HDP.
One possible solution is to increase the "Check Count", or to just disable the alert.