SNMP alert creation through Script and Troubleshooting

[#1]

This article is related to creating SNMP alerts through the custom script and how to troubleshoot.

1. Install SNMP on sandbox or local environment.

   yum install net-snmp net-snmp-utils net-snmp-libs –y

2. Change the script /etc/snmp/snmptrapd.conf file and include "disableAuthorization yes".

   # Example configuration file for snmptrapd
   #
   # No traps are handled by default, you must edit this file!
   #
   # authCommunity   log,execute,net public
   # traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold
   disableAuthorization yes

   To understand why this change required, Refer to the Access Control section of this link.

3. Copy APACHE-AMBARI-MIB.txt file to /usr/share/snmp/mibs folder.
   

   cp /var/lib/ambari-server/resources/APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs

   Note: Ensure it has proper permission

4. Startup a simple SNMP trap daemon to log traps to the /tmp/traps.log file for testing purposes.
   

   nohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log &​

5. Invoke a test trap to ensure that the snmptrapd is logging appropriately to /tmp/traps.log and the Apache Ambari MIB is being respected.
   

   snmptrap -v 2c -c public localhost '' APACHE-AMBARI-MIB::apacheAmbariAlert alertDefinitionName s "definitionName" alertDefinitionHash s "definitionHash" alertName s "name" alertText s "text" alertState i 0 alertHost s "host" alertService s "service" alertComponent s "component" ​

6. You should be able to see the following traps in /tmp/traps.log.
   

   2019-02-04 06:24:38 UDP: [127.0.0.1]:59238->[127.0.0.1]:162 [UDP: [127.0.0.1]:59238->[127.0.0.1]:162]:
   DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (224395199) 25 days, 23:19:11.99SNMPv2-MIB::snmpTrapOID.0 = OID: APACHE-AMBARI-MIB::apacheAmbariAlertAPACHE-AMBARI-MIB::alertDefinitionName = STRING: "definitionName"APACHE-AMBARI-MIB::alertDefinitionHash = STRING: "definitionHash"APACHE-AMBARI-MIB::alertName = STRING: "name"APACHE-AMBARI-MIB::alertText = STRING: "text"APACHE-AMBARI-MIB::alertState = INTEGER: ok(0)APACHE-AMBARI-MIB::alertHost = STRING: "host"APACHE-AMBARI-MIB::alertService = STRING: "service"APACHE-AMBARI-MIB::alertComponent = STRING: "component"​

7. Now, we will be creating the script that will be used by Ambari for sending SNMP traps. Create a file that contains the script, named /tmp/snmp_mib_script.sh. in this example. It is recommended to create this file in a more permanent directory for actual use.
   Format of the Alert script:
   

   #!/bin/bash
   HOST=localhost
   COMMUNITY=public
   STATE=0
   if [[ $4 == "OK" ]]; then
   STATE=0
   elif [[ $4 == "UNKNOWN" ]]; then
   STATE=1
   elif [[ $4 == "WARNING" ]]; then
   STATE=2
   elif [[ $4 == "CRITICAL" ]]; then
   STATE=3
   fi
   /usr/bin/snmptrap -v 2c \
   -c $COMMUNITY $HOST '' APACHE-AMBARI-MIB::apacheAmbariAlert \
   alertDefinitionId i 0 \
   alertDefinitionName s "$1" \
   alertDefinitionHash s "n/a" \
   alertName s "$2" \
   alertText s "$5" \
   alertState i $STATE \
   alertHost s `hostname` \
   alertService s "$3"  ​

   Note: Ensure to change the host with the desired sandbox or host where you want to send the traps.

   Reference: ambari-commits mailing list archives

8. Add the following line to the /etc/ambari-server/conf/ambari.properties file:
   

   org.apache.ambari.contrib.snmp.script=/tmp/snmp_mib_script.sh​

9. Restart the Ambari-server
10. Now, we will use the following API call to add an alert target for the script.
    

    curl -u "admin_user":"admin_password" -H 'X-Requested-By: ambari' http://<Ambari_host>:<PORT>/api/v1/alert_targets -d '{ "AlertTarget" : { "name" : "SNMP_MIB", "description" : "SNMP MIB Target", "notification_type" : "ALERT_SCRIPT","global": true,"properties": {"ambari.dispatch-property.script": "org.apache.ambari.contrib.snmp.script"}}}]}'​

    If you want to update the alert_states as critical only, you can add below lines in script.
    

    { 
    "AlertTarget": { 
    "alert_states": ["CRITICAL"] 
    } 
    } ​

11. You can check the alert notification that has been created in Ambari UI.
    Note: Check if any snmptrapd process is currently running. If so, cancel that process.
    Example:
    

    # ps -ef|grep snmptrapd
    root     15286 15087  0 06:49 pts/0    00:00:00 grep --color=auto snmptrapd
    root     21610     1  0 Jan24 ?        00:00:59 snmptrapd -m ALL -A -n -Lf /tmp/traps.log  
    
    
    #kill -9 21610​

Troubleshooting

1. Whenever any alert will be triggered you will be able to see below lines in Ambari-server.log.
   

   INFO [AlertNoticeDispatchService RUNNING] AlertNoticeDispatchService:279 - There are xx pending alert notices about to be dispatched...​

   This means Ambari is successfully sending the SNMP traps to the mentioned host.

2. At the same timestamp, you can check the entry in ambari-alert.log file.
3. For any change in alert state, there will be an entry that will get recorded in the alert_notice table in the database. If there is no entry available, check the notification set in alert_target table.

Hope this article will help!!!!!

Reference: https://github.com/apache/ambari/tree/trunk/contrib/alert-snmp-mib

[#2]

Great stuff!!!!

[#3]

has anyone implemented the same on cloudera rather than Ambari?