kafka broker restarting all the time

[#1]

hi all

we have hadoop cluster version - 2.6.4 , and separate kafka machines ( 3 kafka machines )

on of the kafka machines ( kafka03 ) is restarting all the time and in some time could be stooped

from netstat we can see that kafka cant listening to port 6667

we check the /var/log/kafka.err file , but this file is empty after we clean it by cp /dev/null /var/log/kafka.err

and start the kafka broker to see if any info will comes to /var/log/kafka.err

any other ideas how to understand why kafka machine not listening to port 6667?

76.12.76.73 is the IP of kafka03 server
76.12.76.74 is the IP of kafka01 server

netstat -tnlpa  | grep 6667
tcp6   0    0 76.12.76.73:43612    76.12.76.74:6667   ESTABLISHED 94962/java
tcp6   0    0 76.12.76.73:43616    76.12.76.74:6667   ESTABLISHED 94962/java
tcp6   0    0 76.12.76.73:43613    76.12.76.74:6667   ESTABLISHED 94962/java 


netstat -tnlpa  | grep 6667
tcp6   0    0 76.12.76.73:43616    76.12.76.74:6667   ESTABLISHED 94962/java
tcp6   0    0 76.12.76.73:43613    76.12.76.74:6667   ESTABLISHED 94962/java 

[#2]

@Michael Bronson

What is the IP of the kafka02?

Create directories

Create data and log directories for ZooKeeper and Kafka. To simplify this process we can add the directories within the user home directory.

In a production environment, we would use different locations, e.g. separate mount points or physical disks for data and log directories.

$ mkdir -p /home/kafka/zookeeper/data 
$ mkdir -p /home/kafka/kafka/kafka-logs

ZooKeeper configuration for embedded

• Share your zookeeper.properties.
  

Set the value to point to the new ZooKeeper directory created above.

dataDir=/home/kafka/zookeeper/data

At the end of this file add all available ZooKeeper servers.

server.1=kafka01:2888:3888
server.2=kafka02:2888:3888
server.3=kafka03:2888:3888

Each in your cluster nodes needs a unique server id. ZooKeeper looks up this information from the following file in my case: /home/kafka/zookeeper/data/myid. You will have to execute a command like this on each server - using a different value for each instance. For instance 1 on server kafka01 we use the value "1".

$ echo "1" > /home/kafka/zookeeper/data/myid

Kafka

• server.properties.
  

Each Kafka cluster node needs a unique id. We have to find the broker.id property in the configuration files and change the id for each server.

broker.id=1

Make sure you have a unique id for your all your brokers broker.id ie 1,2 ,3

Did you change the log directory location specified in the log.dirs parameter?

log.dirs=

Update the listeners and advertised.listeners properties with the current Kafka node hostname.

• listeners: the address/server name and protocol kafka is listening to (internal traffic between Kafka nodes)
• advertised.listener: the address/server name and protocol clients can use to connect to the Kafka cluster (external traffic). Only need to be specified if different from above setting.

i.e

listeners=PLAINTEXT://kafka01:9092
advertised.listeners=PLAINTEXT://kafka01:9092

Next step is to tell Kafka which ZooKeeper nodes can be used to connect to.

zookeeper.connect=kafka01:2181,kafka02:2181,kafka03:2181

Hope that helps you resolve the reboot problem

[#3]

@Geoffrey just note - the machine isnt rebooted ! , kafka broker is restating - its means that from ambari GUI you can see that kafka broker is up / down and finnaly down , but kafka linux machine stay up without reboot

[#4]

@Michael Bronson

Yes I understand there should be some conflict in the config, that's the reason I want you to validate the above parameters.

[#5]

@Geoffrey thank you I will verify this in the next days , and I will update , this is very strange because this happens only on one kafka , while other two kafka's machine are ok

[#6]

@Michael Bronson

You may want to check the logs(server.log) when kafka process stops before changing any existing configs.