Created on 02-04-2019 07:08 AM
This article related to the creating SNMP alert through custom script and How to troubleshoot.
1- Install SNMP on sandbox or local environment.
yum install net-snmp net-snmp-utils net-snmp-libs –y
2- Change the script /etc/snmp/snmptrapd.conf file and include "disableAuthorization yes"
# Example configuration file for snmptrapd # # No traps are handled by default, you must edit this file! # # authCommunity log,execute,net public # traphandle SNMPv2-MIB::coldStart /usr/bin/bin/my_great_script cold disableAuthorization yes
To understand why this change required, Refer the Access Control section of the provided link. -- http://www.net-snmp.org/docs/man/snmptrapd.conf.html
3- Copy APACHE-AMBARI-MIB.txt file to /usr/share/snmp/mibs folder.
cp /var/lib/ambari-server/resources/APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs
*Make Sure it has proper Permission*
4- Startup a simple SNMP trap daemon to log traps to the /tmp/traps.log file for testing purposes.
nohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log &
5- Invoke a test trap to ensure that the snmptrapd is logging appropriately to /tmp/traps.log and the Apache Ambari MIB is being respected.
snmptrap -v 2c -c public localhost '' APACHE-AMBARI-MIB::apacheAmbariAlert alertDefinitionName s "definitionName" alertDefinitionHash s "definitionHash" alertName s "name" alertText s "text" alertState i 0 alertHost s "host" alertService s "service" alertComponent s "component"
6- You should be able to see the below traps in /tmp/traps.log.
2019-02-04 06:24:38 UDP: [127.0.0.1]:59238->[127.0.0.1]:162 [UDP: [127.0.0.1]:59238->[127.0.0.1]:162]: DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (224395199) 25 days, 23:19:11.99SNMPv2-MIB::snmpTrapOID.0 = OID: APACHE-AMBARI-MIB::apacheAmbariAlertAPACHE-AMBARI-MIB::alertDefinitionName = STRING: "definitionName"APACHE-AMBARI-MIB::alertDefinitionHash = STRING: "definitionHash"APACHE-AMBARI-MIB::alertName = STRING: "name"APACHE-AMBARI-MIB::alertText = STRING: "text"APACHE-AMBARI-MIB::alertState = INTEGER: ok(0)APACHE-AMBARI-MIB::alertHost = STRING: "host"APACHE-AMBARI-MIB::alertService = STRING: "service"APACHE-AMBARI-MIB::alertComponent = STRING: "component"
7- Now we will be creating the script which will be used by Ambari for sending SNMP traps. Create a file that contains the script, named /tmp/snmp_mib_script.sh, in this example. It's recommended to create this file in a more permanent directory for actual use.
Format of the Alert Script--
#!/bin/bash HOST=localhost COMMUNITY=public STATE=0 if [[ $4 == "OK" ]]; then STATE=0 elif [[ $4 == "UNKNOWN" ]]; then STATE=1 elif [[ $4 == "WARNING" ]]; then STATE=2 elif [[ $4 == "CRITICAL" ]]; then STATE=3 fi /usr/bin/snmptrap -v 2c \ -c $COMMUNITY $HOST '' APACHE-AMBARI-MIB::apacheAmbariAlert \ alertDefinitionId i 0 \ alertDefinitionName s "$1" \ alertDefinitionHash s "n/a" \ alertName s "$2" \ alertText s "$5" \ alertState i $STATE \ alertHost s `hostname` \ alertService s "$3" \ alertComponent s
Note-- Make Sure to change the Host with the desired sandbox or host where you want to send the traps.
Reference-- http://mail-archives.apache.org/mod_mbox/ambari-commits/201510.mbox/%3C2cdb030d3ef64fffb2540097f885f...
8- Add the following line to the /etc/ambari-server/conf/ambari.properties file
org.apache.ambari.contrib.snmp.script=/tmp/snmp_mib_script.sh
9- Restart the Ambari-server.
10- Now we will use the below api call to add an alert target for the script.
curl -u "admin_user":"admin_password" -H 'X-Requested-By: ambari' http://<Ambari_host>:<PORT>/api/v1/alert_targets -d '{ "AlertTarget" : { "name" : "SNMP_MIB", "description" : "SNMP MIB Target", "notification_type" : "ALERT_SCRIPT","global": true,"properties": {"ambari.dispatch-property.script": "org.apache.ambari.contrib.snmp.script"}}}]}'
If you want to update the alert_states as critical only, you can add below lines in script.
{ "AlertTarget": { "alert_states": ["CRITICAL"] } }
11- You can check the Alert notification that has been created in Ambari UI.
Note-- Check any snmptrapd process is currently running. If so, Cancel that process.
Example--
# ps -ef|grep snmptrapd root 15286 15087 0 06:49 pts/0 00:00:00 grep --color=auto snmptrapd root 21610 1 0 Jan24 ? 00:00:59 snmptrapd -m ALL -A -n -Lf /tmp/traps.log #kill -9 21610
TroubleShooting---
1- Whenever any alert will be triggered you will be able to see below lines in Ambari-server.log.
INFO [AlertNoticeDispatchService RUNNING] AlertNoticeDispatchService:279 - There are xx pending alert notices about to be dispatched...
This means Ambari is successfully sending the SNMP traps to the mentioned host.
2- At the same-time stamp you can check the entry in ambari-alert.log file.
3- For any change in alert state, there will be a entry which will get recorded in alert_notice table in database. If there is no entry available then check the notification set in alert_target table.
Hope this article will help!!!!!
Reference-- https://github.com/apache/ambari/tree/trunk/contrib/alert-snmp-mib
Created on 02-04-2019 09:24 AM
Great stuff!!!!
Created on 09-16-2019 07:00 AM
has anyone implemented the same on cloudera rather than Ambari?
User | Count |
---|---|
758 | |
379 | |
316 | |
309 | |
268 |