Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
New Contributor

This article related to the creating SNMP alert through custom script and How to troubleshoot.

1- Install SNMP on sandbox or local environment.

yum install net-snmp net-snmp-utils net-snmp-libs –y

2- Change the script /etc/snmp/snmptrapd.conf file and include "disableAuthorization yes"

 # Example configuration file for snmptrapd
#
# No traps are handled by default, you must edit this file!
#
# authCommunity   log,execute,net public
# traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold
disableAuthorization yes 

To understand why this change required, Refer the Access Control section of the provided link. -- http://www.net-snmp.org/docs/man/snmptrapd.conf.html

3- Copy APACHE-AMBARI-MIB.txt file to /usr/share/snmp/mibs folder.

 cp /var/lib/ambari-server/resources/APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs 

*Make Sure it has proper Permission*

4- Startup a simple SNMP trap daemon to log traps to the /tmp/traps.log file for testing purposes.

 nohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log & 

5- Invoke a test trap to ensure that the snmptrapd is logging appropriately to /tmp/traps.log and the Apache Ambari MIB is being respected.

 snmptrap -v 2c -c public localhost '' APACHE-AMBARI-MIB::apacheAmbariAlert alertDefinitionName s "definitionName" alertDefinitionHash s "definitionHash" alertName s "name" alertText s "text" alertState i 0 alertHost s "host" alertService s "service" alertComponent s "component" 

6- You should be able to see the below traps in /tmp/traps.log.

2019-02-04 06:24:38 UDP: [127.0.0.1]:59238->[127.0.0.1]:162 [UDP: [127.0.0.1]:59238->[127.0.0.1]:162]:
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (224395199) 25 days, 23:19:11.99SNMPv2-MIB::snmpTrapOID.0 = OID: APACHE-AMBARI-MIB::apacheAmbariAlertAPACHE-AMBARI-MIB::alertDefinitionName = STRING: "definitionName"APACHE-AMBARI-MIB::alertDefinitionHash = STRING: "definitionHash"APACHE-AMBARI-MIB::alertName = STRING: "name"APACHE-AMBARI-MIB::alertText = STRING: "text"APACHE-AMBARI-MIB::alertState = INTEGER: ok(0)APACHE-AMBARI-MIB::alertHost = STRING: "host"APACHE-AMBARI-MIB::alertService = STRING: "service"APACHE-AMBARI-MIB::alertComponent = STRING: "component"

7- Now we will be creating the script which will be used by Ambari for sending SNMP traps. Create a file that contains the script, named /tmp/snmp_mib_script.sh, in this example. It's recommended to create this file in a more permanent directory for actual use.

Format of the Alert Script--

 #!/bin/bash
HOST=localhost
COMMUNITY=public
STATE=0
if [[ $4 == "OK" ]]; then
STATE=0
elif [[ $4 == "UNKNOWN" ]]; then
STATE=1
elif [[ $4 == "WARNING" ]]; then
STATE=2
elif [[ $4 == "CRITICAL" ]]; then
STATE=3
fi
/usr/bin/snmptrap -v 2c \
-c $COMMUNITY $HOST '' APACHE-AMBARI-MIB::apacheAmbariAlert \
alertDefinitionId i 0 \
alertDefinitionName s "$1" \
alertDefinitionHash s "n/a" \
alertName s "$2" \
alertText s "$5" \
alertState i $STATE \
alertHost s `hostname` \
alertService s "$3" \
alertComponent s 

Note-- Make Sure to change the Host with the desired sandbox or host where you want to send the traps.

Reference-- http://mail-archives.apache.org/mod_mbox/ambari-commits/201510.mbox/%3C2cdb030d3ef64fffb2540097f885f...

8- Add the following line to the /etc/ambari-server/conf/ambari.properties file

 org.apache.ambari.contrib.snmp.script=/tmp/snmp_mib_script.sh 

9- Restart the Ambari-server.

10- Now we will use the below api call to add an alert target for the script.

curl -u "admin_user":"admin_password" -H 'X-Requested-By: ambari' http://<Ambari_host>:<PORT>/api/v1/alert_targets -d '{ "AlertTarget" : { "name" : "SNMP_MIB", "description" : "SNMP MIB Target", "notification_type" : "ALERT_SCRIPT","global": true,"properties": {"ambari.dispatch-property.script": "org.apache.ambari.contrib.snmp.script"}}}]}'

If you want to update the alert_states as critical only, you can add below lines in script.

{ 
"AlertTarget": { 
"alert_states": ["CRITICAL"] 
} 
} 

11- You can check the Alert notification that has been created in Ambari UI.

Note-- Check any snmptrapd process is currently running. If so, Cancel that process.

Example--

# ps -ef|grep snmptrapd
root     15286 15087  0 06:49 pts/0    00:00:00 grep --color=auto snmptrapd
root     21610     1  0 Jan24 ?        00:00:59 snmptrapd -m ALL -A -n -Lf /tmp/traps.log  


#kill -9 21610 

TroubleShooting---

1- Whenever any alert will be triggered you will be able to see below lines in Ambari-server.log.

INFO [AlertNoticeDispatchService RUNNING] AlertNoticeDispatchService:279 - There are xx pending alert notices about to be dispatched...

This means Ambari is successfully sending the SNMP traps to the mentioned host.

2- At the same-time stamp you can check the entry in ambari-alert.log file.

3- For any change in alert state, there will be a entry which will get recorded in alert_notice table in database. If there is no entry available then check the notification set in alert_target table.

Hope this article will help!!!!!

Reference-- https://github.com/apache/ambari/tree/trunk/contrib/alert-snmp-mib

309 Views
Comments
Not applicable

Great stuff!!!!

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎02-04-2019 07:08 AM
Updated by:
 
Contributors
Top Kudoed Authors