Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Kafka is running but Ambari shows it is in INSTALLED state

avatar
Rising Star

I am running into a weird issue. I Have 3 brokers installed on 3 different nodes. I assigned to each one a different broker ID.

When I start the Kafka service, the 3 brokers start running, however one of my brokers turns into the red state. ambari-server.logs shows that the Kafka broker turned into INSTALLED state briefly after starting the broker on that host. When I run ps -ef on that host:

ps -ef | grep kafka:

kafka     8670     1  2 18:03 ?        00:00:10 /usr/jdk64/jdk1.8.0_60/bin/java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/var/log/kafka -Dlog4j.configuration=file:/usr/hdp/2.4.0.0-169/kafka/bin/../config/log4j.properties -cp :/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/hdp/2.4.0.0-169/kafka/bin/../libs/* kafka.Kafka /usr/hdp/2.4.0.0-169/kafka/config/server.properties

This means that Kafka is actually running, but Ambari says not!

PS: The kafka pid is well specified through the Kafka PID dir property in the Kafka config section in Ambari.

Any idea ?

1 ACCEPTED SOLUTION

avatar
Rising Star

@Ali Gouta

What is the HDP stack version you are using? You don't need to create Kafka broker ids.

  • Ambari creates broker_id if you are on HDP-2.2
  • Starting in HDP 2.3, Kafka handles the generation of broker.id

Also is the pid in /var/run/kafka/kafka.pid the same as the kafka process that is running?

Do you have kafka-python package installed? If you have you will need to uninstall the kafka-python package.

View solution in original post

7 REPLIES 7

avatar
Master Mentor

what do you mean you assigned broker IDs? Kafka will assign its own IDs, I wouldn't mess with that configuration. I would look in the broker logs for the error message you're having, most likely it's conflict of IDs.

avatar
Rising Star

@Ali Gouta

What is the HDP stack version you are using? You don't need to create Kafka broker ids.

  • Ambari creates broker_id if you are on HDP-2.2
  • Starting in HDP 2.3, Kafka handles the generation of broker.id

Also is the pid in /var/run/kafka/kafka.pid the same as the kafka process that is running?

Do you have kafka-python package installed? If you have you will need to uninstall the kafka-python package.

avatar
Rising Star

You are the best! how did you get the idea of: kafka-python package being installed ? 🙂 Indeed, one of my colleagues installed it :)).Now everyhing is fine. Now, it is clear for me why ambari-agent does not send check_status about kafka.

avatar
Rising Star

Thanks @Ali Gouta 🙂 I had to debug the kafka-python issue in the past.

avatar
Explorer

THANK YOU!!

Removed the kafka-python package and it works as expected.

Going to submit a bug report because of this, there should be no reason for this to happen!

avatar
Explorer

I've submitted a Bug Report because of this, if you are interested to comment : https://issues.apache.org/jira/browse/AMBARI-16778

avatar
Explorer