question Re: Kafka Ambari alert configuration when not using port 6667 in Archives of Support Questions (Read Only)

Kafka Ambari alert configuration when not using port 6667

markdoutre — Sun, 17 Apr 2016 21:39:00 GMT

Currently running HDP 2.3.4.7

Have a Kafka broker running but can't connect to it on port 6667. Turns out this was a switch issue due to port 6667 being blocked. Rather than reconfigure the switch, we've changed to Kafka listener port to 9092.

Everything is working fine but Ambari is raising an alert that it cannot detect that Kafka broker is running. Looking at the alert message, the alert is still trying to connect on port 6667.

Tried restarting Ambari monitoring and Ambari server, but the alert is still picking up port 6667 from somewhere, even though the Kafka confit is set to 9092.

Any ideas? Is 6667 hard coded in somewhere or hidden in a jar file?

Re: Kafka Ambari alert configuration when not using port 6667

jonathanhurley — Sun, 17 Apr 2016 22:38:36 GMT

How did you change the port in Ambari? On HDP 2.3, this alert will look at the kafka-broker/listeners property. In older versions of HDP, it used kafka-broker/port.

Please check what these properties are and set them correctly. When they are not set, the alert defaults to 6667. Chances are it's wanting to use kafka-broker/listeners and that property doesn't exist.

Re: Kafka Ambari alert configuration when not using port 6667

markdoutre — Mon, 18 Apr 2016 02:06:51 GMT

Port was changed in Ambari. Property Kafka-broker/listener set to 9092 at which points Kafka clients could connect to broker.

Re: Kafka Ambari alert configuration when not using port 6667

Former Member — Mon, 18 Apr 2016 02:42:33 GMT

Is it the fresh alert or a Stale alert? Many times we see stale alerts. Which might be generated when the Kafka broker was down for few minutes before the port change.

Re: Kafka Ambari alert configuration when not using port 6667

pminovic — Mon, 18 Apr 2016 12:41:49 GMT

If you still have "kafka-broker/port" can you try setting it to 9092. Since Alerts are a part of Ambari it will be good to know what's your Ambari version, and have you upgraded Ambari recently? Also, please check the file /etc/kakfa/conf/server.properties and make sure "listeners" contain the correct port. In same rare cases it can happen that Ambari doesn't apply actual settings to config files.

Re: Kafka Ambari alert configuration when not using port 6667

markdoutre — Mon, 18 Apr 2016 13:44:25 GMT

Ambari 2.2.0.0. This is a clean install, not an upgrade.

server.properties has listener on 9092

Re: Kafka Ambari alert configuration when not using port 6667

markdoutre — Mon, 18 Apr 2016 13:45:22 GMT

Fresh alerts. Can clear the alert by restarting Ambari, but then it comes back. Looking at the log message, it's trying to connect on 6667

Re: Kafka Ambari alert configuration when not using port 6667

markdoutre — Mon, 18 Apr 2016 18:50:46 GMT

Managed to fix this in the end.

Using the REST API (:8080/api/v1/clusters/XXXXX/alert_definitions/59) I could retrieve the actual alert definition and get the following:

{

  "href" : "http://XXXX:8080/api/v1/clusters/XXXX/alert_definitions/59",
  "AlertDefinition" : {
    "cluster_name" : "XXXXX",
    "component_name" : "KAFKA_BROKER",
    "description" : "This host-level alert is triggered if the Kafka Broker cannot be determined to be up..",
    "enabled" : true,
    "id" : 59,
    "ignore_host" : false,
    "interval" : 1,
    "label" : "Kafka Broker Process",
    "name" : "kafka_broker_process",
    "scope" : "HOST",
    "service_name" : "KAFKA",
    "source" : {
      "default_port" : 6667.0,
      "reporting" : {
        "ok" : {
          "text" : "TCP OK - {0:.3f}s response on port {1}"
        },
        "warning" : {
          "text" : "TCP OK - {0:.3f}s response on port {1}",
          "value" : 1.5
        },
        "critical" : {
          "text" : "Connection failed: {0} to {1}:{2}",
          "value" : 5.0
        }
      },
      "type" : "PORT",
      "uri" : "{{kafka-broker/port}}"
    }
  }
}

The default port is 6667 but it appears to be looking at kafka-broker/port for the actual port address, rather than the listener port set up via Ambari.

Manually changed the default port to 9092 by saving this output to a file, editting and then doing a curl PUT. This changed the alert port and the alert went away.

Re: Kafka Ambari alert configuration when not using port 6667

jonathanhurley — Mon, 18 Apr 2016 19:51:20 GMT

Yes, this is why I suggested you check both kafka-broker/port and kafka-broker/listeners. Changing the default port is fine for now, but if you ever change the port again, you'll need to repeat this step. Instead, it's better to either set kafka-broker/port or change the alert to use kafka-broke/listeners.

I'm guessing that at one point this was an HDP 2.2 cluster (which used kafka-broker/port originally) and then it was upgraded to an HDP 2.3 cluster.

Re: Kafka Ambari alert configuration when not using port 6667

markdoutre — Mon, 18 Apr 2016 21:04:13 GMT

So change "uri":"{{kafka-broker/port}}" to "uri":"{{kafka-broker/listeners}}" I assume?

FYI - Just checked on an untouched 2.4 sandbox and the alert is looking at kafka-broker/port

Re: Kafka Ambari alert configuration when not using port 6667

jonathanhurley — Tue, 19 Apr 2016 01:11:48 GMT

Yes, that should also work. You'll want to PUT the entire JSON back, not just the 1 changed line.

How interesting - that seems like a bug that it's still using the older value on clean 2.4 installs.

Re: Kafka Ambari alert configuration when not using port 6667

madhukanala — Wed, 05 Oct 2016 21:20:57 GMT

even after I updated port number 6667.0 to 9092 and upload the alert.jason file still I see the port number is coming as Float(9092.0),I am using hdp2.3 and below are the json files before and after upload.

still I am abserving this alert in ambari:

Connection failed: [Errno 111] Connection refused to hostname:9092

before: updated port number 6667.0 to 9092 and updated uri from

"uri" : "{{kafka-broker/port}}" to "uri" : "{{kafka-broker/listeners}}"

{
  "AlertDefinition" : {
    "cluster_name" : "hdp",
    "component_name" : "KAFKA_BROKER",
    "description" : "This host-level alert is triggered if the Kafka Broker cannot be determined to be up.",
    "enabled" : true,
    "id" : 101,
    "ignore_host" : false,
    "interval" : 1,
    "label" : "Kafka Broker Process",
    "name" : "kafka_broker_process",
    "scope" : "HOST",
    "service_name" : "KAFKA",
    "source" : {
      "default_port" : 9092,
      "reporting" : {
        "critical" : {
          "value" : 5.0,
          "text" : "Connection failed: {0} to {1}:{2}"
        },
        "warning" : {
          "text" : "TCP OK - {0:.3f}s response on port {1}",
          "value" : 1.5
        },
        "ok" : {
          "text" : "TCP OK - {0:.3f}s response on port {1}"
        }
      },
      "type" : "PORT",
      "uri" : "{{kafka-broker/listeners}}"
    }
  }
}

after i upload this update json file using the below curl command:

curl -u admin:admin -H 'X-Requested-By: ambari' -X PUT "http://awedva3uttpm01.cloud.tiaa-cref.org:8080/api/v1/clusters/hdp/alert_definitions/101" -d @alerts.json

json file after upload:

here my questions is how can i send the port number as integer rather than float.

also checked Ambari logs here it is it doesnt complain about float data type:

05 Oct 2016 10:01:37,178 INFO [qtp-client-24] StackAdvisorRunner:71 - advisor script stderr: 05 Oct 2016 10:01:38,390 ERROR [qtp-client-1225] MetricsPropertyProvider:185 - Error getting timeline metrics. Can not connect to collector, socket error. 05 Oct 2016 10:01:38,740 ERROR [qtp-client-719] MetricsPropertyProvider:185 - Error getting timeline metrics. Can not connect to collector, socket error. 05 Oct 2016 10:01:51,248 INFO [qtp-client-24] PersistKeyValueService:82 - Looking for keyName time-range-service-KAFKA 05 Oct 2016 10:02:59,497 INFO [qtp-client-24] PersistKeyValueService:82 - Looking for keyName mainAlertInstancesController-pagination-displayLength-admin 05 Oct 2016 10:03:09,278 INFO [qtp-client-1225] PersistKeyValueService:82 - Looking for keyName mainHostAlertsController-pagination-displayLength-admin 05 Oct 2016 10:03:16,632 ERROR [qtp-client-23] MetricsPropertyProvider:185 - Error getting timeline metrics. Can not connect to collector, socket error. 05 Oct 2016 10:03:22,145 INFO [qtp-ambari-agent-1466] HeartBeatHandler:657 - State of service component METRICS_MONITOR of service AMBARI_METRICS of cluster hdp has changed from STARTED to INSTALLED at host awedva3uttpm02.cloud.tiaa-cref.org 05 Oct 2016 10:04:24,188 INFO [qtp-ambari-agent-1466] HeartBeatHandler:657 - State of service component METRICS_MONITOR of service AMBARI_METRICS of cluster hdp has changed from INSTALLED to STARTED at host awedva3uttpm02.cloud.tiaa-cref.org

{
  "href" : "http://awedva3uttpm01.cloud.tiaa-cref.org:8080/api/v1/clusters/hdp/alert_definitions/101",
  "AlertDefinition" : {
    "cluster_name" : "hdp",
    "component_name" : "KAFKA_BROKER",
    "description" : "This host-level alert is triggered if the Kafka Broker cannot be determined to be up.",
    "enabled" : true,
    "id" : 101,
    "ignore_host" : false,
    "interval" : 1,
    "label" : "Kafka Broker Process",
    "name" : "kafka_broker_process",
    "scope" : "HOST",
    "service_name" : "KAFKA",
    "source" : {
      "default_port" : 9092.0,
      "reporting" : {
        "critical" : {
          "value" : 5.0,
          "text" : "Connection failed: {0} to {1}:{2}"
        },
        "warning" : {
          "text" : "TCP OK - {0:.3f}s response on port {1}",
          "value" : 1.5
        },
        "ok" : {
          "text" : "TCP OK - {0:.3f}s response on port {1}"
        }
      },
      "type" : "PORT",
      "uri" : "{{kafka-broker/listeners}}"
    }
  }
}

Re: Kafka Ambari alert configuration when not using port 6667

madhukanala — Thu, 06 Oct 2016 02:03:04 GMT

I had to resolve this issue by removing KAFKA service and component from Ambari and adding again from Amabri Server and I have used below Curl commands to update API.

// Get service info

curl-uadmin:admin-H"X-Requested-by:ambari"-i-k-XGET http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KAFKA

// Stop component KAFKA_BROKER

curl-uadmin:admin-H"X-Requested-by:ambari"-i-k-XPUT -d'{"ServiceComponentInfo": {"state": "INSTALLED"}}' http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KNOX/components/KAFKA_BROKER

// Stop service KAFKA

curl-uadmin:admin-H"X-Requested-by:ambari"-i-k-XPUT -d'{"ServiceInfo": {"state": "INSTALLED"}}' http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KAFKA

// Delete component

curl-uadmin:admin-H"X-Requested-by:ambari"-i-k-XDELETE http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/KAFKA_BROKER

// Delete service

curl-uadmin:admin-H"X-Requested-by:ambari"-i-k-XDELETE http://<HOST>:8080/api/v1/clusters/<CLUSTER_NAME>/services/KAFKA

Thanks for all your help

Madhu