Support Questions
Find answers, ask questions, and share your expertise

Kafka is running but Ambari shows it is in INSTALLED state

Solved Go to solution

Kafka is running but Ambari shows it is in INSTALLED state

Contributor

I am running into a weird issue. I Have 3 brokers installed on 3 different nodes. I assigned to each one a different broker ID.

When I start the Kafka service, the 3 brokers start running, however one of my brokers turns into the red state. ambari-server.logs shows that the Kafka broker turned into INSTALLED state briefly after starting the broker on that host. When I run ps -ef on that host:

ps -ef | grep kafka:

kafka     8670     1  2 18:03 ?        00:00:10 /usr/jdk64/jdk1.8.0_60/bin/java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/var/log/kafka -Dlog4j.configuration=file:/usr/hdp/2.4.0.0-169/kafka/bin/../config/log4j.properties -cp :/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/hdp/2.4.0.0-169/kafka/bin/../libs/* kafka.Kafka /usr/hdp/2.4.0.0-169/kafka/config/server.properties

This means that Kafka is actually running, but Ambari says not!

PS: The kafka pid is well specified through the Kafka PID dir property in the Kafka config section in Ambari.

Any idea ?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Kafka is running but Ambari shows it is in INSTALLED state

Contributor

@Ali Gouta

What is the HDP stack version you are using? You don't need to create Kafka broker ids.

  • Ambari creates broker_id if you are on HDP-2.2
  • Starting in HDP 2.3, Kafka handles the generation of broker.id

Also is the pid in /var/run/kafka/kafka.pid the same as the kafka process that is running?

Do you have kafka-python package installed? If you have you will need to uninstall the kafka-python package.

View solution in original post

7 REPLIES 7

Re: Kafka is running but Ambari shows it is in INSTALLED state

Mentor

what do you mean you assigned broker IDs? Kafka will assign its own IDs, I wouldn't mess with that configuration. I would look in the broker logs for the error message you're having, most likely it's conflict of IDs.

Re: Kafka is running but Ambari shows it is in INSTALLED state

Contributor

@Ali Gouta

What is the HDP stack version you are using? You don't need to create Kafka broker ids.

  • Ambari creates broker_id if you are on HDP-2.2
  • Starting in HDP 2.3, Kafka handles the generation of broker.id

Also is the pid in /var/run/kafka/kafka.pid the same as the kafka process that is running?

Do you have kafka-python package installed? If you have you will need to uninstall the kafka-python package.

View solution in original post

Re: Kafka is running but Ambari shows it is in INSTALLED state

Contributor

You are the best! how did you get the idea of: kafka-python package being installed ? :) Indeed, one of my colleagues installed it :)).Now everyhing is fine. Now, it is clear for me why ambari-agent does not send check_status about kafka.

Re: Kafka is running but Ambari shows it is in INSTALLED state

Contributor

Thanks @Ali Gouta :) I had to debug the kafka-python issue in the past.

Re: Kafka is running but Ambari shows it is in INSTALLED state

Explorer

THANK YOU!!

Removed the kafka-python package and it works as expected.

Going to submit a bug report because of this, there should be no reason for this to happen!

Re: Kafka is running but Ambari shows it is in INSTALLED state

Explorer

I've submitted a Bug Report because of this, if you are interested to comment : https://issues.apache.org/jira/browse/AMBARI-16778

Re: Kafka is running but Ambari shows it is in INSTALLED state

Explorer