Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari custom alert failing

Solved Go to solution
Highlighted

Ambari custom alert failing

Super Guru

I have created a custom kafka ambari alert. I have verified I am able to connect via jconsole to view the kafka jmx metrics

The error on ambari alert is:

[Alert][kafka_broker_mean_rate5] Unable to extract JSON from JMX response

My alerts.json:

{
  "AlertDefinition" : {
      "cluster_name" : "kafka-wiz",
      "component_name" : "KAFKA_BROKER",
      "description" : "This kafka broker alert is triggered if input bytes mean rate exeeds 25.",
      "enabled" : true,
      "help_url" : null,
      "ignore_host" : false,
      "interval" : 1,
      "label" : "[CUSTOM6] Kafka Broker Alert",
      "name" : "kafka_broker_mean_rate6",
      "repeat_tolerance" : 1,
      "repeat_tolerance_enabled" : false,
      "scope" : "ANY",
      "service_name" : "KAFKA",
      "source" : {
        "jmx" : {
          "property_list" : ["kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec/MeanRate"],
          "value" : "{0}"
        },
        "reporting" : {
          "ok" : {
            "text" : "Kafka input mean rate: {0}"
          },
          "warning" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 25
          },
          "critical" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 50
          },
          "units" : "bytes"
        },
        "type" : "METRIC",
        "uri" : { "http" : "xxxx.field.xxx.com",
	"default_port" : 16667 }
      }
    }
}

Any feedback appreciated.


2018-07-12-15-41-27.png
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Ambari custom alert failing

Super Guru

I found the issue. Ambari alerts support JMX but only via HTTP. Having JMX alone does not allow for ambari alert to be created. Therefore, what I am trying to do is create ambari alert on kafka where jmx is available through jconsole connnection. However, those metrics are not expose via http and return json result. Basically, it can't be done unless JMX is exposed via HTTP

View solution in original post

3 REPLIES 3
Highlighted

Re: Ambari custom alert failing

Rising Star

@sunile.manjee was looking at the code in Ambari below specific to metric alerts and it seems as if that default_port value is not being used as in other alert types

https://github.com/apache/ambari/blob/branch-2.6/ambari-agent/src/main/python/ambari_agent/alerts/me...

Function used to extract uri appears to only use the default port if uri isn't defined: https://github.com/apache/ambari/blob/branch-2.6/ambari-agent/src/main/python/ambari_agent/alerts/ba...

Also in other examples I've seen the uri is parameterized with a value from Ambari configuration. And when looking at the ambari config the port value is actually included with the uri and not separated out . See example in
Create Custom Alert JSON file" section here: https://community.hortonworks.com/articles/143762/how-to-create-a-custom-ambari-alert-and-use-it-for... .

It may be worthwhile to try including the the port value with the uri



Highlighted

Re: Ambari custom alert failing

Super Guru

@Yolanda M. Davis Thanks for your response. I have tried 7 different combos the the ambari alert and all seem to fail. Here is the ambari alert without default port and the port included in the uri

{
  "AlertDefinition" : {
      "cluster_name" : "kafka-wiz",
      "component_name" : "KAFKA_BROKER",
      "description" : "This kafka broker alert is triggered if input bytes mean rate exeeds 25.",
      "enabled" : true,
      "help_url" : null,
      "ignore_host" : false,
      "interval" : 1,
      "label" : "[CUSTOM7] Kafka Broker Alert",
      "name" : "kafka_broker_mean_rate7",
      "repeat_tolerance" : 1,
      "repeat_tolerance_enabled" : false,
      "scope" : "ANY",
      "service_name" : "KAFKA",
      "source" : {
        "jmx" : {
          "property_list" : ["kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec/MeanRate"],
          "value" : "{0}"
        },
        "reporting" : {
          "ok" : {
            "text" : "Kafka input mean rate: {0}"
          },
          "warning" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 25
          },
          "critical" : {
            "text" : "Kafka input mean rate: {0}",
            "value" : 50
          },
          "units" : "bytes"
        },
        "type" : "METRIC",
        "uri" : { "http" : "kafka-xxx-messaging-1.xxx.xxxxx.com:16667"}
      }
    }
}

I verified I am able to connect via jconsole on that port.

Highlighted

Re: Ambari custom alert failing

Super Guru

I found the issue. Ambari alerts support JMX but only via HTTP. Having JMX alone does not allow for ambari alert to be created. Therefore, what I am trying to do is create ambari alert on kafka where jmx is available through jconsole connnection. However, those metrics are not expose via http and return json result. Basically, it can't be done unless JMX is exposed via HTTP

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here