I recently upgraded HDP and kafka was one of the component.
HDP Version : 2.4
Kafka - 0.9.0.2
The upgrade was fine and smooth. After the upgrade and finalize, producer and consumers were failing. After debugging I found that the broker.ids are changed to 1001 from 1.
Before upgrade broker.ids are 1,2,3
after upgrade , broker.ids are 1001,1002,1003.
The problem is all the topics are pointing to old broker.ids.
[kafka@usw2stdpka01 ~]$ /usr/hdp/2.4.0.0-169/kafka/bin/kafka-topics.sh --zookeeper usw2stdpzo01:2181 --describe --topic search_log
Topic:search_log PartitionCount:10 ReplicationFactor:3 Configs:
Topic: search_log Partition: 0 Leader: 0 Replicas: 1,2,0 Isr: 0
Topic: search_log Partition: 1 Leader: 0 Replicas: 2,0,1 Isr: 0
I created a test topic and it is pointing to new broker_ids
[kafka@usw2stdpka01 ~]$ /usr/hdp/2.4.0.0-169/kafka/bin/kafka-topics.sh --zookeeper usw2stdpzo01:2181 --describe --topic test
Topic:test PartitionCount:10 ReplicationFactor:2 Configs:
Topic: test Partition: 0 Leader: 1001 Replicas: 1001,1002 Isr: 1001,1002
Topic: test Partition: 1 Leader: 1002 Replicas: 1002,1003 Isr: 1002,1003
Changing the topics with new broker id is not possible and it is time consuming. How do I change it back to original broker_ids. I couldn't find the broker_id info in the server.properties or in ambari console config. This results in failure of producer and consumer messages.
