Support Questions
Find answers, ask questions, and share your expertise

Kafka partitions reassignment - how to rebalance partial partitions of the topic

Lets say i want to re-balance the topic - `cars_data_hibrid_machines` on my `kafka` cluster

 

We have 6 kafka brokers - they are part of ambari cluster with hdp - 2.6.5 , 

 

topic - `cars_data_hibrid_machines` include `200` partitions , and replication `factor is 3`

In that case I will create the following `car.json`

{
"version": 1,
"topics": [
{ "topic": "cars_data_hibrid_machines"
]
}

Then

We run the following

kafka-reassign-partitions.sh --zookeeper $ZK_HOSTS --topics-to-move-json-file car.json --broker-list "1,2,3,4,5,6" --generate

and

kafka-reassign-partitions.sh --zookeeper $ZK_HOSTS --reassignment-json-file reassignment.json –execute

 

now lets say we want to re balance only the first `20 partitions` of the topic - `cars_data_hibrid_machines` on my kafka cluster

**in that case how to build the json files with topics partitions ?**

 

example of the topic partitions

Topic: cars_data_hibrid_machines Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5,2,1
Topic: cars_data_hibrid_machines Partition: 1 Leader: 6 Replicas: 6,2,3 Isr: 6,2,3
Topic: cars_data_hibrid_machines Partition: 2 Leader: 1 Replicas: 1,3,4 Isr: 4,1,3
Topic: cars_data_hibrid_machines Partition: 3 Leader: 2 Replicas: 2,4,5 Isr: 4,5,2
Topic: cars_data_hibrid_machines Partition: 4 Leader: 3 Replicas: 3,5,6 Isr: 5,6,3
Topic: cars_data_hibrid_machines Partition: 5 Leader: 4 Replicas: 4,6,1 Isr: 4,6,1

.
.
.
.
.
.

Michael-Bronson
1 REPLY 1

Re: Kafka partitions reassignment - how to rebalance partial partitions of the topic

Cloudera Employee

Hi Michael,

 

Rebalancing just the first 20 partitions of a table is similar to balancing all the partitions, but the only difference is you need to specify 2o partitions alone in the JSON file.