I am running into a weird issue. I Have 3 brokers installed on 3 different nodes. I assigned to each one a different broker ID. When I start the Kafka service, the 3 brokers start running, however one of my brokers turns into the red state. ambari-server.logs shows that the Kafka broker turned into INSTALLED state briefly after starting the broker on that host. When I run ps -ef on that host: ps -ef | grep kafka: kafka 8670 1 2 18:03 ? 00:00:10 /usr/jdk64/jdk1.8.0_60/bin/java -Xmx1G -Xms1G -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/data/var/log/kafka/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dkafka.logs.dir=/data/var/log/kafka -Dlog4j.configuration=file:/usr/hdp/2.4.0.0-169/kafka/bin/../config/log4j.properties -cp :/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/lib/ambari-metrics-kafka-sink/ambari-metrics-kafka-sink.jar:/usr/lib/ambari-metrics-kafka-sink/lib/*:/usr/hdp/2.4.0.0-169/kafka/bin/../libs/* kafka.Kafka /usr/hdp/2.4.0.0-169/kafka/config/server.properties This means that Kafka is actually running, but Ambari says not! PS: The kafka pid is well specified through the Kafka PID dir property in the Kafka config section in Ambari. Any idea ? ... View more

If a node running ambari-agent reboots or one component (one kafka broker for instance) fails due to some out of memory issue. Is it possible to automatically restart the service without a human intervention ? Supervisord or systemctl may handle this kind of issue. However, with Ambari I am not sure this is supported. I was reading the following jira which looks to be related to this subject: AMBARI-10029 I did the with ambari-2.2.0 test and once the VM was up again, components did not recover automatically. I need to restart them manually from Ambari... I think it is important to have this feature especially in a production environement! ... View more

All my HDP components succeded to start after kerborising my cluster, except Kakfa... This is what I can see in /var/log/kafka/server.log: [2016-01-06 23:04:54,931] FATAL Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.IllegalArgumentException: You must pass java.security.auth.login.config in secure mode. at org.apache.kafka.common.security.kerberos.Login.login(Login.java:297) Ambari logs do not show any error message related to the Kafka fail while restarting the service. I am running HDP 2.3 and Ambari 2.1 Any clue ? ... View more