question Re: How to get rid of stale alerts in Ambari in Support Questions

How to get rid of stale alerts in Ambari

pardeep_kumar — Mon, 19 Aug 2019 12:15:36 GMT

I have restarted Ambari Server and all agents along with complete HDP stack multiple times in past 5 days for different activities but these alerts don't go away.

Re: How to get rid of stale alerts in Ambari

nsabharwal — Thu, 14 Jan 2016 00:28:46 GMT

@Pardeep You may have to open a support ticket for this. I had the same experience and support had to troubleshoot it

Re: How to get rid of stale alerts in Ambari

Mark_Petronic — Tue, 19 Jan 2016 02:15:40 GMT

@Neeraj Sabharwal

Hey Neeraj! I'm having this same issue now, too. Do you have any input regarding how support troubleshoot it for you or do I need to hit them up as well with a ticket for an answer?

Re: How to get rid of stale alerts in Ambari

nsabharwal — Tue, 19 Jan 2016 03:53:44 GMT

@Mark Petronic Support will webex and check for configs and other settings related to AMS.

Re: How to get rid of stale alerts in Ambari

smishra1 — Tue, 19 Jan 2016 05:30:07 GMT

Check the ambari heap size, it may be running out of memory.

/var/lib/ambari-server/ambari-env.sh

Change -Xmx2048m to 8GB if you have enough memory availbale and restart ambari-server.

Re: How to get rid of stale alerts in Ambari

Mark_Petronic — Tue, 19 Jan 2016 10:20:14 GMT

Well, I solved it another way - I just upgraded to Ambari 2.2. I will watch see if this comes back as I run with this version and open a ticket if that happens. I was in the midst of upgrading anyway when this started to happen. I need to move to HDP 2.3.4 and Spark 1.5. But, 8 GB, really? Wow! That seems ridiculously expensive for a monitoring framework.

Re: How to get rid of stale alerts in Ambari

pminovic — Wed, 20 Jan 2016 15:20:57 GMT

Hi @Pardeep with Support's help we got rid of those alerts by adding 'misfire_grace_time':10 to APS_CONFIG in /usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py on every node. After the update that section should read:

APS_CONFIG = { 
'threadpool.core_threads': 3, 
'coalesce': True, 
'standalone': False, 
'misfire_grace_time':10 
}

In this we are allowing up to 10 seconds for all tests to complete. After that restart all ambari_agents. We tried on one cluster and it worked. This is most likely fixed in Ambari-2.2 but happens in 2.1.2.

Re: How to get rid of stale alerts in Ambari

sushil61 — Fri, 05 Feb 2016 09:40:28 GMT

HDP 2.3, Ambari 2.1

Adding 'misfire_grace_time':10 to APS_CONFIG in /usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py on every node and restarting ambari server and agent on all nodes didn't worked for me.

Re: How to get rid of stale alerts in Ambari

jeff1 — Tue, 12 Apr 2016 01:06:43 GMT

FWIW, You can disable this alert. Click "Enabled" next to State in upper-right and this alert will no longer be checked.

Re: How to get rid of stale alerts in Ambari

KuldeepK — Tue, 12 Apr 2016 12:48:01 GMT

@Sagar Shimpi

Re: How to get rid of stale alerts in Ambari

openbusinessintelligence — Mon, 02 May 2016 15:23:42 GMT

I have the same issue with an HDP 2.4 on SLES 11.4 fresh install.

Many of the alerts regard timeouts chacking the UIs:

Data Node UI, Node Manager UI, Atrals UI, Oozie UI, etc.

All these URLs are reachable from both the windows laptops with the browser an the HDP-nodes using wget in the console.

How does Ambari check these URLs? Is it possible that the check-scripts ignore os-wide proxy- and firewall-configurations?

I would also like to know what does the "24-Hour" columns mean, does someone have an idea what does it mean? There is no mention at all of this field in the docs. The content is alway "0" in my cluster (like in the posted screenshot).

Many thanks

Re: How to get rid of stale alerts in Ambari

openbusinessintelligence — Mon, 02 May 2016 15:54:09 GMT

@Predrag Minovic

In my Ambari the code look like:

self.APS_CONFIG = {
      'apscheduler.threadpool.core_threads': 3,
      'apscheduler.coalesce': True,
      'apscheduler.standalone': False,
      'apscheduler.misfire_grace_time': alert_grace_period
    }

Re: How to get rid of stale alerts in Ambari

bjorn_olsen — Mon, 15 May 2017 15:21:23 GMT

In Ambari 2.2 misfire_grace_time is configurable by changing the value for the variable alert_grace_period.

This variable is configured in /etc/ambari-agent/conf/ambari-agent.ini and the default is:

alert_grace_period=5

You can increase to 10 seconds to match above answer.

This needs to be done on all hosts running ambari-agent, and afterwards ambari-agent needs to be restarted.

Re: How to get rid of stale alerts in Ambari

kkum — Thu, 15 Jun 2017 04:34:08 GMT

I have tried to place this parameter

APS_CONFIG ={'threadpool.core_threads':3,'coalesce':True,'standalone':False,'misfire_grace_time':10} in the following file /usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py
Changed the parmeter and restarted all ambari-agents but failed with following error

====================================================================================

Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/ambari_agent/AmbariAgent.py", line 24, in <module> from Controller import AGENT_AUTO_RESTART_EXIT_CODE File "/usr/lib/python2.6/site-packages/ambari_agent/Controller.py", line 44, in <module> from ambari_agent.AlertSchedulerHandler import AlertSchedulerHandler File "/usr/lib/python2.6/site-packages/ambari_agent/AlertSchedulerHandler.py", line 50 'misfire_grace_time':10 ^SyntaxError: invalid syntax
=========================================================================================
Please help me out how to fix this issue
Appreciate the help

Re: How to get rid of stale alerts in Ambari

gauravb117 — Thu, 07 Dec 2017 16:25:07 GMT

After increasing the timeout, the error still persists and do you think this might be because of system coonfiguration??

Re: How to get rid of stale alerts in Ambari

DavidR — Thu, 30 Jan 2020 16:20:59 GMT

This was what fixed it for me.

Using Ambari Version 2.7.3.0

SSH into host that's reporting stale alerts:

sudo vi /etc/ambari-agent/conf/ambari-agent.ini

change alert_grace_period (default 5 | change to: 15)

optional: restart service's instance reporting stale alerts (JournalNodes, NameNodes,etc)

sudo systemctl restart ambari-agent.service

Example using CentOS / Redhat

Re: How to get rid of stale alerts in Ambari

VikneshSK — Tue, 11 Aug 2020 08:40:08 GMT

I did this in root user, found the file and changed it there. But, how to change it for each node?