Support Questions
Find answers, ask questions, and share your expertise

ambari warning host disk usage

Contributor

I have a node on my cluster that is getting "host disk usage" alerts in ambari

This node has clients installed but is not a data node.

My thresholds for alerts are set via ambari and /var/lib/ambari-server/resources/host_scripts/alert_disk_space.py for warning at 90% and critical at 95%. The warning is shows usage at 52.9% and disk usage is only 56% (via df)

Any idea how to clear this? It doesn't seem to be impacting things but is annoying

ambari version 2.2.2.0

HDP stack HDP-2.2.0.0-2041

13 REPLIES 13

Rising Star

Multiple things to check here:

1. Was host in question removed and added back to cluster ?

2. Does host in question where the disk alert runs, differs in time compared to Ambari server time clock?

3. Is "host usage alert" actually running on this host ? We may be seeing a stale alert. You may disable and enable the "host usage alert" from Ambari UI.

4. If time is not an issue (Point 2), increase misfire_grace_time in AlertSchedulerHandler.py to 20, and see if that helps.

Let me know, if the issue is still seen.

Contributor

Thanks! disabling and then re-enabling the alert from ambari cleared the problem

Rising Star

@Jon Page Thanks for confirming.

Contributor

Actually I spoke too soon

The alert warning has returned. I verified time is in sync and tried changing the misfire grace time from 5 to 20. Still getting the warning

Rising Star

- Can you "disable" and "enabled after changing the misfire time?

Expert Contributor

Thanks @Swapan Shridhar! #2 did the trick for me!

Super Collaborator

Can you post the text of the alert? There's also a limit on the total amount of free space available on the host.

Contributor

The text of the alert is

Capacity Used:[52.90%, 120.1 GB], Capacity Total: [227.0 GB], path=/usr/hdp

Super Collaborator

That to me sounds like it's still using the default warning value of 50%. How exactly did you change the thresholds and which version of Ambari is this?