Created 11-09-2016 04:36 AM
Hi all,
My ambari server has this alert.
HDFS Storage Capacity Usage (Daily)
I find this alert in this doc but no information for handle it.
This service-level alert is triggered if the increase in storage capacity usage deviation has grown beyond the specified threshold within a day period.
The variance for this alert is 2,015,674,228B which is 73% of the 2,750,867,892B average (1,375,433,946B is the limit)
Anyone can help to resolve this issue.?
Created 11-09-2016 05:53 AM
You can set the alert threshold based on your requirement.
Ambari IU --> Alerts (Tab) --> "Alert Definition Filter" Search for "HDFS Storage Capacity Usage" --> Click on "Edit" button
Now you can define a new "Growth Rate" CRITICAL & WARNING thresholds.
Also if your HDFS has some unwanted data in the ".trash" then you should clean it. Please see: https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html#expunge
Also you should check your HDFS filesystem usage to validate the alert is correct or False.
# su - hdfs # hdfs dfs -df -h Filesystem Size Used Available Use% hdfs://My-NN-HA 310.5 G 8.9 G 200.0 G 3%
.
Created on 11-09-2016 06:10 AM - edited 08-18-2019 03:46 AM
I flow you to check HDFS Storage Capacity Usage in Alert tab.
and it have Default setting:
so in my first post
is that the number 2,015,674,228B is Byte used by HDFS average daily?
and if true so how i change limit 1,375,433,946B ?
Created 04-17-2017 08:36 AM
@Hoang Le have you been able to overcome this issue?
I am having the same alerts and the alert is always there.
I haven't started using HDFS, haven't put and files either. I would like to know the reason if you found out. Also I formatted hdfs manually. Before this format, alerts were not critical.
Created 04-17-2017 08:44 AM
you can increase growth rate and capacity to avoid this alert.
Created 05-10-2017 10:36 AM
Formatting hdfs is a critical task and requires the correct approach. In my case I formatted namenodes as journal nodes down. This led to consistency problems of journal quorum, nn1 and nn2. txids, editlog were different. For this reason alert was critical.
Created 01-07-2017 10:10 AM
I have changed the Growth rate but still I am getting the warning. can anyone help me storage-growth-rate.png
after the changes after-growth-rate-changes.png
,
Created on 04-10-2018 12:09 PM - edited 08-18-2019 03:46 AM
Maybe it is reasonable to get this alert when starting a new cluster, but annoying if you are just testing how is HDP.
One possible solution is to increase the "Check Count", or to just disable the alert.