- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Kafka is running but Ambari shows it is in INSTALLED state
- Labels:
-
Apache Ambari
-
Apache Kafka
Created 04-19-2016 04:16 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am running into a weird issue. I Have 3 brokers installed on 3 different nodes. I assigned to each one a different broker ID.
When I start the Kafka service, the 3 brokers start running, however one of my brokers turns into the red state. ambari-server.logs shows that the Kafka broker turned into INSTALLED state briefly after starting the broker on that host. When I run ps -ef on that host:
ps -ef | grep kafka: kafka 8670 1 2 18:03 ? 00:00:10 /usr/jdk64/jdk1.8.0_60/bin/java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/var/log/kafka -Dlog4j.configuration=file:/usr/hdp/2.4.0.0-169/kafka/bin/../config/log4j.properties -cp :/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/hdp/2.4.0.0-169/kafka/bin/../libs/* kafka.Kafka /usr/hdp/2.4.0.0-169/kafka/config/server.properties
This means that Kafka is actually running, but Ambari says not!
PS: The kafka pid is well specified through the Kafka PID dir property in the Kafka config section in Ambari.
Any idea ?
Created 04-19-2016 05:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What is the HDP stack version you are using? You don't need to create Kafka broker ids.
- Ambari creates broker_id if you are on HDP-2.2
- Starting in HDP 2.3, Kafka handles the generation of broker.id
Also is the pid in /var/run/kafka/kafka.pid the same as the kafka process that is running?
Do you have kafka-python package installed? If you have you will need to uninstall the kafka-python package.
Created 04-19-2016 05:10 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
what do you mean you assigned broker IDs? Kafka will assign its own IDs, I wouldn't mess with that configuration. I would look in the broker logs for the error message you're having, most likely it's conflict of IDs.
Created 04-19-2016 05:26 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What is the HDP stack version you are using? You don't need to create Kafka broker ids.
- Ambari creates broker_id if you are on HDP-2.2
- Starting in HDP 2.3, Kafka handles the generation of broker.id
Also is the pid in /var/run/kafka/kafka.pid the same as the kafka process that is running?
Do you have kafka-python package installed? If you have you will need to uninstall the kafka-python package.
Created 04-19-2016 05:59 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You are the best! how did you get the idea of: kafka-python package being installed ? 🙂 Indeed, one of my colleagues installed it :)).Now everyhing is fine. Now, it is clear for me why ambari-agent does not send check_status about kafka.
Created 04-19-2016 06:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks @Ali Gouta 🙂 I had to debug the kafka-python issue in the past.
Created 05-19-2016 05:30 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
THANK YOU!!
Removed the kafka-python package and it works as expected.
Going to submit a bug report because of this, there should be no reason for this to happen!
Created 05-19-2016 05:44 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I've submitted a Bug Report because of this, if you are interested to comment : https://issues.apache.org/jira/browse/AMBARI-16778
Created 06-09-2016 02:18 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Ambari has resolved the issue : https://git1-us-west.apache.org/repos/asf?p=ambari.git&a=commit&h=93624db6d7740b8d877af77978328bc724...