question Ambari custom alert failing in Archives of Support Questions (Read Only)

Ambari custom alert failing

sunile_manjee — Fri, 13 Jul 2018 03:48:46 GMT

I have created a custom kafka ambari alert. I have verified I am able to connect via jconsole to view the kafka jmx metrics

The error on ambari alert is:

[Alert][kafka_broker_mean_rate5] Unable to extract JSON from JMX response

My alerts.json:

{
  "AlertDefinition" : {
      "cluster_name" : "kafka-wiz",
      "component_name" : "KAFKA_BROKER",
      "description" : "This kafka broker alert is triggered if input bytes mean rate exeeds 25.",
      "enabled" : true,
      "help_url" : null,
      "ignore_host" : false,
      "interval" : 1,
      "label" : "[CUSTOM6] Kafka Broker Alert",
      "name" : "kafka_broker_mean_rate6",
      "repeat_tolerance" : 1,
      "repeat_tolerance_enabled" : false,
      "scope" : "ANY",
      "service_name" : "KAFKA",
      "source" : {
        "jmx" : {
          "property_list" : ["kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec/MeanRate"],
          "value" : "{0}"
        },
        "reporting" : {
          "ok" : {
            "text" : "Kafka input mean rate: {0}"
          },
          "warning" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 25
          },
          "critical" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 50
          },
          "units" : "bytes"
        },
        "type" : "METRIC",
        "uri" : { "http" : "xxxx.field.xxx.com",
	"default_port" : 16667 }
      }
    }
}

Any feedback appreciated.

Re: Ambari custom alert failing

ydavis — Fri, 13 Jul 2018 09:31:15 GMT

@sunile.manjee was looking at the code in Ambari below specific to metric alerts and it seems as if that default_port value is not being used as in other alert types

https://github.com/apache/ambari/blob/branch-2.6/ambari-agent/src/main/python/ambari_agent/alerts/metric_alert.py#L87-L110

Function used to extract uri appears to only use the default port if uri isn't defined: https://github.com/apache/ambari/blob/branch-2.6/ambari-agent/src/main/python/ambari_agent/alerts/base_alert.py#L331-L389

Also in other examples I've seen the uri is parameterized with a value from Ambari configuration. And when looking at the ambari config the port value is actually included with the uri and not separated out . See example in
Create Custom Alert JSON file" section here: https://community.hortonworks.com/articles/143762/how-to-create-a-custom-ambari-alert-and-use-it-for.html .

It may be worthwhile to try including the the port value with the uri

Re: Ambari custom alert failing

sunile_manjee — Wed, 18 Jul 2018 22:46:38 GMT

@Yolanda M. Davis Thanks for your response. I have tried 7 different combos the the ambari alert and all seem to fail. Here is the ambari alert without default port and the port included in the uri

{
  "AlertDefinition" : {
      "cluster_name" : "kafka-wiz",
      "component_name" : "KAFKA_BROKER",
      "description" : "This kafka broker alert is triggered if input bytes mean rate exeeds 25.",
      "enabled" : true,
      "help_url" : null,
      "ignore_host" : false,
      "interval" : 1,
      "label" : "[CUSTOM7] Kafka Broker Alert",
      "name" : "kafka_broker_mean_rate7",
      "repeat_tolerance" : 1,
      "repeat_tolerance_enabled" : false,
      "scope" : "ANY",
      "service_name" : "KAFKA",
      "source" : {
        "jmx" : {
          "property_list" : ["kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec/MeanRate"],
          "value" : "{0}"
        },
        "reporting" : {
          "ok" : {
            "text" : "Kafka input mean rate: {0}"
          },
          "warning" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 25
          },
          "critical" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 50
          },
          "units" : "bytes"
        },
        "type" : "METRIC",
        "uri" : { "http" : "kafka-xxx-messaging-1.xxx.xxxxx.com:16667"}
      }
    }
}

I verified I am able to connect via jconsole on that port.

Re: Ambari custom alert failing

sunile_manjee — Mon, 23 Jul 2018 23:37:06 GMT

I found the issue. Ambari alerts support JMX but only via HTTP. Having JMX alone does not allow for ambari alert to be created. Therefore, what I am trying to do is create ambari alert on kafka where jmx is available through jconsole connnection. However, those metrics are not expose via http and return json result. Basically, it can't be done unless JMX is exposed via HTTP