Kafka - broker.id changed after HDP upgrade

anand_ranganath — Sun, 19 Jun 2016 07:52:58 GMT

I recently upgraded HDP and kafka was one of the component.

HDP Version : 2.4

Kafka - 0.9.0.2

The upgrade was fine and smooth. After the upgrade and finalize, producer and consumers were failing. After debugging I found that the broker.ids are changed to 1001 from 1.

Before upgrade broker.ids are 1,2,3

after upgrade , broker.ids are 1001,1002,1003.

The problem is all the topics are pointing to old broker.ids.

[kafka@usw2stdpka01 ~]$ /usr/hdp/2.4.0.0-169/kafka/bin/kafka-topics.sh --zookeeper usw2stdpzo01:2181  --describe --topic search_log
Topic:search_log	PartitionCount:10	ReplicationFactor:3	Configs:
	Topic: search_log	Partition: 0	Leader: 0	Replicas: 1,2,0	Isr: 0
	Topic: search_log	Partition: 1	Leader: 0	Replicas: 2,0,1	Isr: 0

I created a test topic and it is pointing to new broker_ids

[kafka@usw2stdpka01 ~]$ /usr/hdp/2.4.0.0-169/kafka/bin/kafka-topics.sh --zookeeper usw2stdpzo01:2181  --describe --topic test
Topic:test	PartitionCount:10	ReplicationFactor:2	Configs:
	Topic: test	Partition: 0	Leader: 1001	Replicas: 1001,1002	Isr: 1001,1002
	Topic: test	Partition: 1	Leader: 1002	Replicas: 1002,1003	Isr: 1002,1003

Changing the topics with new broker id is not possible and it is time consuming. How do I change it back to original broker_ids. I couldn't find the broker_id info in the server.properties or in ambari console config. This results in failure of producer and consumer messages.

Re: Kafka - broker.id changed after HDP upgrade

pminovic — Sun, 19 Jun 2016 07:55:52 GMT

For troubleshooting this and other potential Kafka upgrade issues please see this.

Re: Kafka - broker.id changed after HDP upgrade

anand_ranganath — Sun, 19 Jun 2016 08:21:04 GMT

Thanks @Predrag Minovic, that resolved my problem.

question Kafka - broker.id changed after HDP upgrade in Archives of Support Questions (Read Only)

Kafka - broker.id changed after HDP upgrade

Re: Kafka - broker.id changed after HDP upgrade

Re: Kafka - broker.id changed after HDP upgrade