Apache Kafka partition reassignment reg.

[#1]

I have a Kafka cluster of 3 nodes. I want to add 2 more nodes to the cluster to make it 5 node cluster.
After adding the nodes, I will do a FULL CLUSTER RESTART. i.e., shutdown all kafka & zookeeper services at once and then start all the services in 5 nodes.

I want to know
1. Does the FULL CLUSTER RESTART distribute the topic partitions evenly across 5 nodes?
Or need to run the partition reassignment tool kafka-reassign-partitions.sh ?
2. Doesn't Kafka have internal mechanism to automatically reassign the partitions unless running kafka-reassign-partitions.sh

Please clarify. Thank you.

[#2]

Hello, 

Regarding to your questions:

1. Does the FULL CLUSTER RESTART distribute the topic partitions evenly across 5 nodes?
All the topics already created will remain the same.
Or need to run the partition reassignment tool kafka-reassign-partitions.sh ?
Yes, this is the available approach.

2. Doesn't Kafka have an internal mechanism to automatically reassign the partitions unless running kafka-reassign-partitions.sh
Unfortunately, this task has to be done manually by using the reassign partition tool.

You can check how this tool works by looking at the video article below:

https://www.youtube.com/watch?v=TjKtCKsUjbQ

Hope this helps.

Regards,

Manuel.

[#3]

Thank you Manuel for your response.
I tried the reassign partition tool and seen its not reliable.

• The problem with reassign partition tool kafka-reassign-partitions.sh is: it takes huge time in re-balancing each partition and during this time the entire cluster performance is impacted.
• When I ran this tool, it was stuck with one partition and it hung there for more than a day.
  The Cluster performance was severely impacted, and we had to restart the entire cluster.
• I don't see a way even to stop the tool when its taking long time.

Can you please suggest how to overcome this situations.

[#4]

Hi Vinay,

Do you see any error in logs while running "reassign partition tool" ? This might help to debug issue.

Were all the brokers healthy and ISR were good before you ran the tool?

***When I ran this tool, it was stuck with one partition and it hung there for more than a day.
The Cluster performance was severely impacted, and we had to restart the entire cluster.

>> I can suggest if the data/topics are more then probably you can do reassignment of subset of topics to avoid load on the cluster.
You can provide a list of topics that should be moved to the new set of brokers and a target list of new brokers.

***I don't see a way even to stop the tool when its taking long time.

>> You can abort the assignment by deleting the "/admin/reassign_partitions" zk node on your zookeeper cluster using zookeeper shell, and move the partitions that are assigned to the dead broker to new nodes.

Thanks
Sagar S