SNMP alert creation through Script and Troubleshooting

[#1]

This article related to the creating SNMP alert through custom script and How to troubleshoot.

1- Install SNMP on sandbox or local environment.

yum install net-snmp net-snmp-utils net-snmp-libs –y

2- Change the script /etc/snmp/snmptrapd.conf file and include "disableAuthorization yes"

 # Example configuration file for snmptrapd
#
# No traps are handled by default, you must edit this file!
#
# authCommunity   log,execute,net public
# traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold
disableAuthorization yes 

To understand why this change required, Refer the Access Control section of the provided link. -- http://www.net-snmp.org/docs/man/snmptrapd.conf.html

3- Copy APACHE-AMBARI-MIB.txt file to /usr/share/snmp/mibs folder.

 cp /var/lib/ambari-server/resources/APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs 

*Make Sure it has proper Permission*

4- Startup a simple SNMP trap daemon to log traps to the /tmp/traps.log file for testing purposes.

 nohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log & 

5- Invoke a test trap to ensure that the snmptrapd is logging appropriately to /tmp/traps.log and the Apache Ambari MIB is being respected.

 snmptrap -v 2c -c public localhost '' APACHE-AMBARI-MIB::apacheAmbariAlert alertDefinitionName s "definitionName" alertDefinitionHash s "definitionHash" alertName s "name" alertText s "text" alertState i 0 alertHost s "host" alertService s "service" alertComponent s "component" 

6- You should be able to see the below traps in /tmp/traps.log.

2019-02-04 06:24:38 UDP: [127.0.0.1]:59238->[127.0.0.1]:162 [UDP: [127.0.0.1]:59238->[127.0.0.1]:162]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (224395199) 25 days, 23:19:11.99SNMPv2-MIB::snmpTrapOID.0 = OID: APACHE-AMBARI-MIB::apacheAmbariAlertAPACHE-AMBARI-MIB::alertDefinitionName = STRING: "definitionName"APACHE-AMBARI-MIB::alertDefinitionHash = STRING: "definitionHash"APACHE-AMBARI-MIB::alertName = STRING: "name"APACHE-AMBARI-MIB::alertText = STRING: "text"APACHE-AMBARI-MIB::alertState = INTEGER: ok(0)APACHE-AMBARI-MIB::alertHost = STRING: "host"APACHE-AMBARI-MIB::alertService = STRING: "service"APACHE-AMBARI-MIB::alertComponent = STRING: "component"

7- Now we will be creating the script which will be used by Ambari for sending SNMP traps. Create a file that contains the script, named /tmp/snmp_mib_script.sh, in this example. It's recommended to create this file in a more permanent directory for actual use.

Format of the Alert Script--

 #!/bin/bash
HOST=localhost
COMMUNITY=public
STATE=0
if [[ $4 == "OK" ]]; then
STATE=0
elif [[ $4 == "UNKNOWN" ]]; then
STATE=1
elif [[ $4 == "WARNING" ]]; then
STATE=2
elif [[ $4 == "CRITICAL" ]]; then
STATE=3
fi
/usr/bin/snmptrap -v 2c \
-c $COMMUNITY $HOST '' APACHE-AMBARI-MIB::apacheAmbariAlert \
alertDefinitionId i 0 \
alertDefinitionName s "$1" \
alertDefinitionHash s "n/a" \
alertName s "$2" \
alertText s "$5" \
alertState i $STATE \
alertHost s `hostname` \
alertService s "$3" \
alertComponent s 

Note-- Make Sure to change the Host with the desired sandbox or host where you want to send the traps.

Reference-- http://mail-archives.apache.org/mod_mbox/ambari-commits/201510.mbox/%3C2cdb030d3ef64fffb2540097f885f...

8- Add the following line to the /etc/ambari-server/conf/ambari.properties file

 org.apache.ambari.contrib.snmp.script=/tmp/snmp_mib_script.sh 

9- Restart the Ambari-server.

10- Now we will use the below api call to add an alert target for the script.

curl -u "admin_user":"admin_password" -H 'X-Requested-By: ambari' http://<Ambari_host>:<PORT>/api/v1/alert_targets -d '{ "AlertTarget" : { "name" : "SNMP_MIB", "description" : "SNMP MIB Target", "notification_type" : "ALERT_SCRIPT","global": true,"properties": {"ambari.dispatch-property.script": "org.apache.ambari.contrib.snmp.script"}}}]}'

If you want to update the alert_states as critical only, you can add below lines in script.

{ 
"AlertTarget": { 
"alert_states": ["CRITICAL"] 
} 
} 

11- You can check the Alert notification that has been created in Ambari UI.

Note-- Check any snmptrapd process is currently running. If so, Cancel that process.

Example--

# ps -ef|grep snmptrapd
root     15286 15087  0 06:49 pts/0    00:00:00 grep --color=auto snmptrapd
root     21610     1  0 Jan24 ?        00:00:59 snmptrapd -m ALL -A -n -Lf /tmp/traps.log  


#kill -9 21610 

TroubleShooting---

1- Whenever any alert will be triggered you will be able to see below lines in Ambari-server.log.

INFO [AlertNoticeDispatchService RUNNING] AlertNoticeDispatchService:279 - There are xx pending alert notices about to be dispatched...

This means Ambari is successfully sending the SNMP traps to the mentioned host.

2- At the same-time stamp you can check the entry in ambari-alert.log file.

3- For any change in alert state, there will be a entry which will get recorded in alert_notice table in database. If there is no entry available then check the notification set in alert_target table.

Hope this article will help!!!!!

Reference-- https://github.com/apache/ambari/tree/trunk/contrib/alert-snmp-mib

[#2]

Great stuff!!!!

[#3]

has anyone implemented the same on cloudera rather than Ambari?