Community Articles
Find and share helpful community-sourced technical articles
Labels (1)
Cloudera Employee

This article is related to creating SNMP alerts through the custom script and how to troubleshoot.

  1. Install SNMP on sandbox or local environment.
    yum install net-snmp net-snmp-utils net-snmp-libs –y
  2. Change the script /etc/snmp/snmptrapd.conf file and include "disableAuthorization yes".
    # Example configuration file for snmptrapd
    #
    # No traps are handled by default, you must edit this file!
    #
    # authCommunity   log,execute,net public
    # traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold
    disableAuthorization yes

    To understand why this change required, Refer to the Access Control section of this link.

  3. Copy APACHE-AMBARI-MIB.txt file to /usr/share/snmp/mibs folder.

    cp /var/lib/ambari-server/resources/APACHE-AMBARI-MIB.txt /usr/share/snmp/mibs

    Note: Ensure it has proper permission

  4. Startup a simple SNMP trap daemon to log traps to the /tmp/traps.log file for testing purposes.
    nohup snmptrapd -m ALL -A -n -Lf /tmp/traps.log &​
  5. Invoke a test trap to ensure that the snmptrapd is logging appropriately to /tmp/traps.log and the Apache Ambari MIB is being respected.
    snmptrap -v 2c -c public localhost '' APACHE-AMBARI-MIB::apacheAmbariAlert alertDefinitionName s "definitionName" alertDefinitionHash s "definitionHash" alertName s "name" alertText s "text" alertState i 0 alertHost s "host" alertService s "service" alertComponent s "component" ​
  6. You should be able to see the following traps in /tmp/traps.log.
    2019-02-04 06:24:38 UDP: [127.0.0.1]:59238->[127.0.0.1]:162 [UDP: [127.0.0.1]:59238->[127.0.0.1]:162]:
    DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (224395199) 25 days, 23:19:11.99SNMPv2-MIB::snmpTrapOID.0 = OID: APACHE-AMBARI-MIB::apacheAmbariAlertAPACHE-AMBARI-MIB::alertDefinitionName = STRING: "definitionName"APACHE-AMBARI-MIB::alertDefinitionHash = STRING: "definitionHash"APACHE-AMBARI-MIB::alertName = STRING: "name"APACHE-AMBARI-MIB::alertText = STRING: "text"APACHE-AMBARI-MIB::alertState = INTEGER: ok(0)APACHE-AMBARI-MIB::alertHost = STRING: "host"APACHE-AMBARI-MIB::alertService = STRING: "service"APACHE-AMBARI-MIB::alertComponent = STRING: "component"​
  7. Now, we will be creating the script that will be used by Ambari for sending SNMP traps. Create a file that contains the script, named /tmp/snmp_mib_script.sh. in this example. It is recommended to create this file in a more permanent directory for actual use.
    Format of the Alert script:
    #!/bin/bash
    HOST=localhost
    COMMUNITY=public
    STATE=0
    if [[ $4 == "OK" ]]; then
    STATE=0
    elif [[ $4 == "UNKNOWN" ]]; then
    STATE=1
    elif [[ $4 == "WARNING" ]]; then
    STATE=2
    elif [[ $4 == "CRITICAL" ]]; then
    STATE=3
    fi
    /usr/bin/snmptrap -v 2c \
    -c $COMMUNITY $HOST '' APACHE-AMBARI-MIB::apacheAmbariAlert \
    alertDefinitionId i 0 \
    alertDefinitionName s "$1" \
    alertDefinitionHash s "n/a" \
    alertName s "$2" \
    alertText s "$5" \
    alertState i $STATE \
    alertHost s `hostname` \
    alertService s "$3"  ​

    Note: Ensure to change the host with the desired sandbox or host where you want to send the traps.

    Reference: ambari-commits mailing list archives

  8. Add the following line to the /etc/ambari-server/conf/ambari.properties file:
    org.apache.ambari.contrib.snmp.script=/tmp/snmp_mib_script.sh​
  9. Restart the Ambari-server
  10. Now, we will use the following API call to add an alert target for the script.
    curl -u "admin_user":"admin_password" -H 'X-Requested-By: ambari' http://<Ambari_host>:<PORT>/api/v1/alert_targets -d '{ "AlertTarget" : { "name" : "SNMP_MIB", "description" : "SNMP MIB Target", "notification_type" : "ALERT_SCRIPT","global": true,"properties": {"ambari.dispatch-property.script": "org.apache.ambari.contrib.snmp.script"}}}]}'​
    If you want to update the alert_states as critical only, you can add below lines in script.
    { 
    "AlertTarget": { 
    "alert_states": ["CRITICAL"] 
    } 
    } ​
  11. You can check the alert notification that has been created in Ambari UI.
    Note: Check if any snmptrapd process is currently running. If so, cancel that process.
    Example:
    # ps -ef|grep snmptrapd
    root     15286 15087  0 06:49 pts/0    00:00:00 grep --color=auto snmptrapd
    root     21610     1  0 Jan24 ?        00:00:59 snmptrapd -m ALL -A -n -Lf /tmp/traps.log  
    
    
    #kill -9 21610​

Troubleshooting

  1. Whenever any alert will be triggered you will be able to see below lines in Ambari-server.log.
    INFO [AlertNoticeDispatchService RUNNING] AlertNoticeDispatchService:279 - There are xx pending alert notices about to be dispatched...​

    This means Ambari is successfully sending the SNMP traps to the mentioned host.

  2. At the same timestamp, you can check the entry in ambari-alert.log file.
  3. For any change in alert state, there will be an entry that will get recorded in the alert_notice table in the database. If there is no entry available, check the notification set in alert_target table.

Hope this article will help!!!!!

Reference: https://github.com/apache/ambari/tree/trunk/contrib/alert-snmp-mib

1,714 Views
Comments
Cloudera Employee

Great stuff!!!!

New Contributor

has anyone implemented the same on cloudera rather than Ambari?

Don't have an account?
Version history
Last update:
‎02-15-2021 02:20 AM
Updated by:
Contributors
Top Kudoed Authors