Kafka is running but Ambari shows it is in INSTALLED state

[#1]

I am running into a weird issue. I Have 3 brokers installed on 3 different nodes. I assigned to each one a different broker ID.

When I start the Kafka service, the 3 brokers start running, however one of my brokers turns into the red state. ambari-server.logs shows that the Kafka broker turned into INSTALLED state briefly after starting the broker on that host. When I run ps -ef on that host:

ps -ef | grep kafka:

kafka     8670     1  2 18:03 ?        00:00:10 /usr/jdk64/jdk1.8.0_60/bin/java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/var/log/kafka -Dlog4j.configuration=file:/usr/hdp/2.4.0.0-169/kafka/bin/../config/log4j.properties -cp :/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/hdp/2.4.0.0-169/kafka/bin/../libs/* kafka.Kafka /usr/hdp/2.4.0.0-169/kafka/config/server.properties

This means that Kafka is actually running, but Ambari says not!

PS: The kafka pid is well specified through the Kafka PID dir property in the Kafka config section in Ambari.

Any idea ?

[#2]

what do you mean you assigned broker IDs? Kafka will assign its own IDs, I wouldn't mess with that configuration. I would look in the broker logs for the error message you're having, most likely it's conflict of IDs.

[#3 · Accepted solution]

@Ali Gouta

What is the HDP stack version you are using? You don't need to create Kafka broker ids.

• Ambari creates broker_id if you are on HDP-2.2
• Starting in HDP 2.3, Kafka handles the generation of broker.id

Also is the pid in /var/run/kafka/kafka.pid the same as the kafka process that is running?

Do you have kafka-python package installed? If you have you will need to uninstall the kafka-python package.

[#4]

You are the best! how did you get the idea of: kafka-python package being installed ? 🙂 Indeed, one of my colleagues installed it :)).Now everyhing is fine. Now, it is clear for me why ambari-agent does not send check_status about kafka.

[#5]

Thanks @Ali Gouta 🙂 I had to debug the kafka-python issue in the past.

[#6]

THANK YOU!!

Removed the kafka-python package and it works as expected.

Going to submit a bug report because of this, there should be no reason for this to happen!

[#7]

I've submitted a Bug Report because of this, if you are interested to comment : https://issues.apache.org/jira/browse/AMBARI-16778

[#8]

Ambari has resolved the issue : https://git1-us-west.apache.org/repos/asf?p=ambari.git&a=commit&h=93624db6d7740b8d877af77978328bc724...