Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

unable to start kafka brokers in cloudera

unable to start kafka brokers in cloudera

HI,

 

I am using Cloudera CDH 5.14.4

 

I have added csd file and then downloaded and actiavted kafka parcel. After adding the kafka service I am unable to start the kafka brokers.

 

csd: kafka-1.2.0.jar
Kafka parcel: 2.0.2-1.2.0.2.p0.5

 

I found below error in logfile /var/log/kafka/kafka-broker-hostnameXXX.log

 

2018-08-22 16:27:07,747 INFO kafka.server.KafkaServer: starting
2018-08-22 16:27:07,754 INFO kafka.server.KafkaServer: Connecting to zookeeper on b2brp-cdh-cedn0.hostanameXXXXXX:2181,b2brp-cdh-cedn1.hostanameXXXXXX:2181,b2brp-cdh-cmsn0.hostanameXXXXXX:2181,b2brp-cdh-cmsn1.hostanameXXXXXX:2181,b2brp-cdh-cmsn2.hostanameXXXXXX:2181
2018-08-22 16:27:07,755 INFO org.I0Itec.zkclient.ZkClient: JAAS File name: /run/cloudera-scm-agent/process/1160-kafka-KAFKA_BROKER/jaas.conf
2018-08-22 16:27:07,757 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=b2brp-cdh-cedn0.hostanameXXXXXX:2181,b2brp-cdh-cedn1.hostanameXXXXXX:2181,b2brp-cdh-cmsn0.hostanameXXXXXX:2181,b2brp-cdh-cmsn1.hostanameXXXXXX:2181,b2brp-cdh-cmsn2.hostanameXXXXXX:2181 sessionTimeout=6000 watcher=org.I0Itec.zkclient.ZkClient@7a220c9a
2018-08-22 16:27:07,757 INFO org.I0Itec.zkclient.ZkEventThread: Starting ZkClient event thread.
2018-08-22 16:27:07,759 INFO org.I0Itec.zkclient.ZkClient: Waiting for keeper state SaslAuthenticated
2018-08-22 16:27:07,764 INFO org.apache.zookeeper.client.ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism.
2018-08-22 16:27:07,765 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server b2brp-cdh-cmsn1.hostanameXXXXXX/xx.xx.xx.xx:2181. Will attempt to SASL-authenticate using Login Context section 'Client'
2018-08-22 16:27:07,766 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to b2brp-cdh-cmsn1.hostanameXXXXXX/xx.xx.xx.xx:2181, initiating session
2018-08-22 16:27:07,776 INFO org.apache.zookeeper.ClientCnxn: Session establishment complete on server b2brp-cdh-cmsn1.hostanameXXXXXX/xx.xx.xx.xx:2181, sessionid = 0x564e90a3c9c9b85, negotiated timeout = 6000
2018-08-22 16:27:07,776 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SyncConnected)
2018-08-22 16:27:07,784 INFO org.I0Itec.zkclient.ZkClient: zookeeper state changed (SaslAuthenticated)
2018-08-22 16:27:07,830 INFO kafka.log.LogManager: Loading logs.
2018-08-22 16:27:07,837 INFO kafka.log.LogManager: Logs loading complete.
2018-08-22 16:27:08,108 INFO kafka.log.LogManager: Starting log cleanup with a period of 300000 ms.
2018-08-22 16:27:08,110 INFO kafka.log.LogManager: Starting log flusher with a default period of 9223372036854775807 ms.
2018-08-22 16:27:08,112 INFO kafka.log.LogCleaner: Starting the log cleaner
2018-08-22 16:27:08,117 INFO kafka.log.LogCleaner: [kafka-log-cleaner-thread-0], Starting
2018-08-22 16:27:08,124 FATAL kafka.server.KafkaServer: Fatal error during KafkaServer startup. Prepare to shutdown
kafka.common.InconsistentBrokerIdException: Configured broker.id 341 doesn't match stored broker.id 186 in
 meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to
 create a new broker, you should remove all data in your data directories (log.dirs).
        at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:635)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:184)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
        at kafka.Kafka$.main(Kafka.scala:67)
        at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
        at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
2018-08-22 16:27:08,126 INFO kafka.server.KafkaServer: shutting down

2018-08-22 16:27:08,129 INFO kafka.log.LogManager: Shutting down.
2018-08-22 16:27:08,130 INFO kafka.log.LogCleaner: Shutting down the log cleaner.
2018-08-22 16:27:08,130 INFO kafka.log.LogCleaner: [kafka-log-cleaner-thread-0], Shutting down
2018-08-22 16:27:08,131 INFO kafka.log.LogCleaner: [kafka-log-cleaner-thread-0], Stopped
2018-08-22 16:27:08,131 INFO kafka.log.LogCleaner: [kafka-log-cleaner-thread-0], Shutdown completed
2018-08-22 16:27:08,137 INFO kafka.log.LogManager: Shutdown complete.
2018-08-22 16:27:08,138 INFO org.I0Itec.zkclient.ZkEventThread: Terminate ZkClient event thread.
2018-08-22 16:27:08,147 INFO org.apache.zookeeper.ZooKeeper: Session: 0x564e90a3c9c9b85 closed
2018-08-22 16:27:08,147 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2018-08-22 16:27:08,149 INFO kafka.server.KafkaServer: shut down completed
2018-08-22 16:27:08,150 FATAL kafka.server.KafkaServerStartable: Fatal error during KafkaServerStartable startup. Prepare to shutdown
kafka.common.InconsistentBrokerIdException: Configured broker.id 341 doesn't match stored broker.id 186 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
        at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:635)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:184)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
        at kafka.Kafka$.main(Kafka.scala:67)
        at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76)
        at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala)
2018-08-22 16:27:08,150 INFO kafka.server.KafkaServer: shutting down

 

Can anyone please help me to resolve this issue.

 

Thanks in advance.

 

1 REPLY 1

Re: unable to start kafka brokers in cloudera

Hi,

 

Issue has resolved and kafka borker is up and running fine now :)

I have modified the broker.id value in  meta.properties(borker_id=341). This changes I did in all kafka broker machines in the path /var/local/kafka/data.

 

Thanks.