Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to purge/reset Ambari alerts

Solved Go to solution

How to purge/reset Ambari alerts

Explorer

Hello I am using Ambari version 2.4.2. Can anyone provide the steps for purging alerts especially those that are stale without having to delete the alert or restart the individual components/services?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to purge/reset Ambari alerts

Super Collaborator

You can clear out an individual alert's state by disabling it and then re-enabling it. This will cause all active instances of that alert to disappear and it will run clean. On some versions of Ambari, this was required when you did things like delete a host which could leave orphaned alerts which never run again (and thus become stale).

If you are seeing the actual "Stale Alert" trigger, you'll want to identify which alerts are causing it to fire - in other words, which alerts are not running. Disabling/Enabling those could help - but if they seem to continue to be stale, then something else is going on which is preventing them from running.

8 REPLIES 8

Re: How to purge/reset Ambari alerts

Super Mentor

@Freemon Johnson

You can try using the "db-cleanup" option (in ambari 2.4 and 2.2.2), It will batch delete the ambari.alert_notice, ambari.alert_current ,ambari.alert_history tablesPrior to the date mentioned.

# ambari-server db-cleanup -d 2017-09-15 --cluster-name=DemoCluster

.

From amabri 2.5 this is more enhanced and changed as "db-purge-history" . The db-purge-history command will analyze mode tables (in addition to onlye the alerts table)

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.2.0/bk_ambari-administration/content/purging-am...

Re: How to purge/reset Ambari alerts

Explorer

@Jay SenSharma

Thank you very much for the reference. I will keep this approach in my notes in case the answer below fails to work.

Re: How to purge/reset Ambari alerts

Super Collaborator

You can clear out an individual alert's state by disabling it and then re-enabling it. This will cause all active instances of that alert to disappear and it will run clean. On some versions of Ambari, this was required when you did things like delete a host which could leave orphaned alerts which never run again (and thus become stale).

If you are seeing the actual "Stale Alert" trigger, you'll want to identify which alerts are causing it to fire - in other words, which alerts are not running. Disabling/Enabling those could help - but if they seem to continue to be stale, then something else is going on which is preventing them from running.

Re: How to purge/reset Ambari alerts

Explorer
@Jonathan Hurley

Thank you Jonathan! Disabling and re-enabling worked just fine.

Re: How to purge/reset Ambari alerts

I have purge some alerts by Enalbe/Disable. but one alert appeared again after Enable/Disable.
How to purge it ? Please see the image :

42971-data-node-healthy.jpg

Thanks

Highlighted

Re: How to purge/reset Ambari alerts

Super Collaborator

Hi Mudassar,

Generally, it's better to open a new issue instead of tacking onto an existing one since the problem/resolution could be very different.

To answer your question, no, you can't clear it in this case. This is a metric alert coming from HDFS. The HDFS service is broadcasting that 1 DataNode is considered dead. Ambari is simply detecting this and alerting on it. You'll need to figure out why the NameNode is sending that metric.

Normally I think the NN considers a DataNode "dead" after more than a few minutes of lost contact (without a decommission). However, if the DataNode makes contact again, it should be clearing it.

Re: How to purge/reset Ambari alerts

Explorer

@Mudassar Hussain

When I posted my initial question it was pertaining to alerts that are triggered because of HDFS storage capacity exceeding a configurable threshold. Then there are some alerts that trigger once a week, month about capacity or if you restarted a service. In short these are trivial alerts that are just annoying but not detrimental to the health of the cluster.

In your image you provided correct me if I am wrong, it appears you have an issue with your namenode. If that is accurate that is serious and that is an alert you do not want to ignore. Have you checked to see if your name node volumes are corrupted? A datanode is corrupted or exceded capacity?

I found this article as well. Not sure if this is precisely related:

https://community.hortonworks.com/questions/81840/node-in-maintenance-mode-throws-stale-alert-from-m...

Re: How to purge/reset Ambari alerts

Thanks @Jonathan Hurley got your point regarding raising question. My @Namenode was on Safemode that time when i was facing this alert. i put the @namenode into Normal mode. now the alert Purge. is this exactly the reason ? whats your views
Thanks

Don't have an account?
Coming from Hortonworks? Activate your account here