Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Alert notification not working

Alert notification not working

New Contributor

I am trying to get an alert notification script to fire without any success. I'm using the HDP sandbox.


I found a few articles - like this one: https://cwiki.apache.org/confluence/display/AMBARI/Creating+a+Script-based+Alert+Dispatcher+-+2.4.0


I'm following the steps. However, I keep seeing errors in the ambari-server.log like:

2019-04-16 18:53:38,751 WARN [script-dispatcher-1] AlertScriptDispatcher$AlertScriptRunnable:363 - Unable to dispatch ALERT_SCRIPT notification because /contrib/ambari-alerts/scripts/scaler-notification.py terminated with exit code 127


My entire Python script consists of a single line:

i = 1+1


Previously, I had a few more statements in there, using the above link as a guide. But basically just taking the args and printing them. I was getting a status code of '2' then.


I am using a global setting and the alert is set up correctly:

notification.dispatch.alert.script=/contrib/ambari-alerts/scripts/scaler-notification.py


I have full permissions (777) on this directory structure so it's not a permissions issue.

My path: /usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/bin/python


I am at a loss as to what could be the issue with this firing correctly. Please let me know what you think.

3 REPLIES 3

Re: Alert notification not working

Super Mentor

@Michael Klaene

I tried the following steps in Ambari 2.6.2 and it worked perfectly fine. Can you please validate the steps and compare it with yours to see if everything is correct?

Step1). Posted the following custom alert notification data to AmbariServer.

# curl -i -u admin:admin -H 'X-Requested-By: ambari' -X POST  "http://newhwx1.example.com:8080/api/v1/alert_targets"  -d '
  {
    "AlertTarget": 
      {
        "name": "test_dispatcher", 
        "description": "Custom Notification Dispatcher", 
        "notification_type": "ALERT_SCRIPT", 
        "global": true, 
        "alert_states": ["CRITICAL","WARNING","UNKNOWN","OK"], 
        "properties": { 
          "ambari.dispatch-property.script": "notification.dispatch.alert.script"
        }
      }
  }
'


Step-2). Edited the "ambari.properties" file and added the following line:

# grep 'notification.dispatch.alert.script' /etc/ambari-server/conf/ambari.properties
notification.dispatch.alert.script=/contrib/ambari-alerts/scripts/scaler-notification.py


Step-3). Wrote the Script "scaler-notification.py" as following:

# cat /contrib/ambari-alerts/scripts/scaler-notification.py


#!/usr/bin/env python
from datetime import datetime
import sys
def test_notification():
  definitionName = sys.argv[1]
  definitionLabel = sys.argv[2]
  serviceName = sys.argv[3]
  alertState = sys.argv[4]
  alertText = sys.argv[5]
  timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
  notification_data = str(timestamp + ' ************* My Alert Dispatcher Logic Here ************' + " -- " + definitionName + " -- " + definitionLabel + " -- " + serviceName + " -- " + alertState + " -- " + alertText + " -- ")

  ## Writing notification to a file. Use your own logic here.
  file = open("/var/log/ambari-server/custom_notification.log", "a+")
  file.write(notification_data)
  file.close()

if __name__ == '__main__':
  test_notification()


Step-4). Changed the permission on the file so that it can have Execute Permissions.

# chmod +x /contrib/ambari-alerts/scripts/scaler-notification.py
# ls -l /contrib/ambari-alerts/scripts/scaler-notification.py
-rwxr-xr-x. 1 root root 537 Apr 17 01:23 /contrib/ambari-alerts/scripts/scaler-notification.py


Step-5). Restarted Ambari Server

# ambari-server restart



Step-6). Stop some component/service so that we get some alerts triggered.



Step-7). I tailed the log file which my Custom Script Notification Dispatcher was writing and i could see the entries as following:

# tail -f /var/log/ambari-server/custom_notification.log

2019-04-17 01:39:00 ************* My Alert Dispatcher Logic Here ************ -- kafka_broker_process -- Kafka Broker Process -- KAFKA -- CRITICAL -- Connection failed: [Errno 111] Connection refused to newhwx3.example.com:6667 --

.

So the Custom Notification worked fine.

Re: Alert notification not working

New Contributor

Your example worked perfectly as described. Here's where my issue was, I'm sure, as everything else was identical:


In your POST to create the target, you had:

"properties": {

"ambari.dispatch-property.script": "notification.dispatch.alert.script"

}


However, my POST looks exactly as it does in the Confluence example:

POST api/v1/alert_targets
    
    {
      "AlertTarget": {
        "name": "syslogger",
        "description": "Syslog Target",
        "notification_type": "ALERT_SCRIPT",
        "global": true
      }
    }


I get that this is the 'glue' that tells it the property to look for in the ambari.properties but is it required for a global target? I assume it is?


Thanks again

Re: Alert notification not working

New Contributor

How did you solve this problem

WARN [script-dispatcher-1] AlertScriptDispatcher$AlertScriptRunnable:363 - Unable to dispatch ALERT_SCRIPT notification because /var/lib/ambari-server/resources/scripts/ambarialter.py terminated with exit code 2

I have this problem now