Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Apache Kafka partition reassignment reg.

avatar
New Contributor

I have a Kafka cluster of 3 nodes. I want to add 2 more nodes to the cluster to make it 5 node cluster.
After adding the nodes, I will do a FULL CLUSTER RESTART. i.e., shutdown all kafka & zookeeper services at once and then start all the services in 5 nodes.

I want to know
1. Does the FULL CLUSTER RESTART distribute the topic partitions evenly across 5 nodes?
Or need to run the partition reassignment tool kafka-reassign-partitions.sh ?
2. Doesn't Kafka have internal mechanism to automatically reassign the partitions unless running kafka-reassign-partitions.sh

Please clarify. Thank you.

3 REPLIES 3

avatar
Expert Contributor

Hello, 

 

Regarding to your questions:

 

1. Does the FULL CLUSTER RESTART distribute the topic partitions evenly across 5 nodes?
All the topics already created will remain the same.
Or need to run the partition reassignment tool kafka-reassign-partitions.sh ?
Yes, this is the available approach.

2. Doesn't Kafka have an internal mechanism to automatically reassign the partitions unless running kafka-reassign-partitions.sh
Unfortunately, this task has to be done manually by using the reassign partition tool.

 

You can check how this tool works by looking at the video article below:

https://www.youtube.com/watch?v=TjKtCKsUjbQ

 

Hope this helps.

Regards,

Manuel.

 

avatar
New Contributor

Thank you Manuel for your response.
I tried the reassign partition tool and seen its not reliable.

  • The problem with reassign partition tool kafka-reassign-partitions.sh is: it takes huge time in re-balancing each partition and during this time the entire cluster performance is impacted.
  • When I ran this tool, it was stuck with one partition and it hung there for more than a day.
    The Cluster performance was severely impacted, and we had to restart the entire cluster.
  • I don't see a way even to stop the tool when its taking long time.

Can you please suggest how to overcome this situations.

avatar
Expert Contributor

Hi Vinay,

 

Do you see any error in logs while running "reassign partition tool" ? This might help to debug issue.

Were all the brokers healthy and ISR were good before you ran the tool?

 

***When I ran this tool, it was stuck with one partition and it hung there for more than a day.
The Cluster performance was severely impacted, and we had to restart the entire cluster.

>> I can suggest if the data/topics are more then probably you can do reassignment of subset of topics to avoid load on the cluster.
You can provide a list of topics that should be moved to the new set of brokers and a target list of new brokers.

 

***I don't see a way even to stop the tool when its taking long time.

>> You can abort the assignment by deleting the "/admin/reassign_partitions" zk node on your zookeeper cluster using zookeeper shell, and move the partitions that are assigned to the dead broker to new nodes.

 

Thanks
Sagar S