Reply
Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Move partitions from invalid leader

Hi ,

I have a situation .

i have 3 broker(id) ==> 58,57,56,

some how broker 57 went invalid 

 

hduser@node01:~$ kafka-topics --zookeeper node01:2181,node02:2181,rac1:2181 --describe

Topic:clouder PartitionCount:1 ReplicationFactor:2 Configs:
Topic: clouder Partition: 0 Leader: 56 Replicas: 56,57 Isr: 56
Topic:truckevent PartitionCount:2 ReplicationFactor:1 Configs:
Topic: truckevent Partition: 0 Leader: -1 Replicas: 57 Isr: 57
Topic: truckevent Partition: 1 Leader: 56 Replicas: 56 Isr: 56

 

 i moved to replica from 57 to 58 

 

root@node01:/opt/TruckEvents# kafka-topics --zookeeper node01:2181,node02:2181,rac1:2181 --describe
Topic:clouder PartitionCount:1 ReplicationFactor:2 Configs:
Topic: clouder Partition: 0 Leader: 56 Replicas: 56,57 Isr: 56
Topic:truckevent PartitionCount:2 ReplicationFactor:2 Configs:
Topic: truckevent Partition: 0 Leader: -1 Replicas: 58,57 Isr: 57
Topic: truckevent Partition: 1 Leader: 56 Replicas: 56 Isr: 56

 

but i also want to move leadership to 58 and tried 

 

root@node01:/opt/TruckEvents# kafka-preferred-replica-election --zookeeper node01:2181,node02:2181,rac1:2181
Successfully started preferred replica election for partitions Set([truckevent,1], [truckevent,0], [clouder,0])

nothing changed and Leader is still 57 .

 

how can we change the leader is its invaid ?

 

 

Expert Contributor
Posts: 113
Registered: ‎02-15-2016

Re: Move partitions from invalid leader

Anyone ? pls help
Cloudera Employee
Posts: 266
Registered: ‎01-09-2014

Re: Move partitions from invalid leader

For this partition:

Topic: truckevent Partition: 0 Leader: -1 Replicas: 57 Isr: 57

 

Since you only have replication-factor of 1, you don't have another replica to bring up as the leader.  In this instance, you can move the replica to a running broker, but you will have to enable unclean leader election, as that new replica isn't in the ISR for partition 0.  You won't have any of the data that is in that partition, as it all resides on broker 57.  If you can move that data to broker 58, you would preserve the messages in that partition.

 

If you need to go that route, you would need to use the kafka-reassign-partitions command to take the single partition (truckevent partition 0 on broker 57) and move it to another broker. See the full documentation here [1]

 

 

In this example, we are moving partition 0 for the testmove topic from broker 52 to 51 in this example:

 

 

 

[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove    PartitionCount:2    ReplicationFactor:2    Configs:
    Topic: testmove    Partition: 0    Leader: -1    Replicas: 52    Isr: 52
    Topic: testmove    Partition: 1    Leader: 50    Replicas: 50    Isr: 50

 

 

1. Create the topictomove.json file:

 

echo '{"topics": [{"topic": "testmove"}], "version":1 }' > topictomove.json

 

 

2. generate the partitions to reassign using the --generate option:

 

root@host-1 ~]# kafka-reassign-partitions --zookeeper ${HOSTNAME}:2181 --generate --broker-list 51,50 --topics-to-move-json-file topictomove.json
Current partition replica assignment

{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[52]},{"topic":"testmove","partition":1,"replicas":[50]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[50]},{"topic":"testmove","partition":1,"replicas":[51]}]}

 

 

3.  Take the output from the "Proposed partition reassignment configuration" and include just the partition you want to move, in a file called reassign.json.  In this instance, we are moving partition0 from replica 52 to replica 51 so the broker id listed in the json file will be the new target:

 

[root@host-1 ~]# cat reassign.json
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[51]}]}

 

4.  If possible, copy the partition folder from the invalid broker to the new broker.  This will ensure the partition contains all the data on the new broker:

 

rsync -ave ssh /var/local/kafka/data/testmove-0 broker51:/var/local/kafka/data/

 

 

5. Use the --execute command to make the changes:

 

[root@host-1 ~]# kafka-reassign-partitions --zookeeper ${HOSTNAME}:2181 --execute --reassignment-json-file reassign.json
Current partition replica assignment

{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[52]},{"topic":"testmove","partition":1,"replicas":[50]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[51]}]}

 

6.  At this time, the new replica will show up, but its not in the ISR list, and it can't be elected leader:

 

[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove    PartitionCount:2    ReplicationFactor:2    Configs:
    Topic: testmove    Partition: 0    Leader: -1    Replicas: 51,52    Isr: 52
    Topic: testmove    Partition: 1    Leader: 50    Replicas: 50    Isr: 50

And in the logs on the new broker, you will see the following:

 

 

2016-08-05 10:45:17,119 ERROR state.change.logger: Broker 51 received LeaderAndIsrRequest with correlation id 126 from controller 50 epoch 4 for partition [testmove,0] but cannot become follower since the new leader -1 is unavailable.

 

7.  You will have to enable unclean.leader.election.enable in Cloudera Manager for the Kafka service configuration, and restart the kafka service.  The broker that is the active controller needs to be restarted with this flag enabled to properly allow the new leader to be elected, even though it is not in the ISR list.

 

7.  The new replica is now shown as the new leader:

 

[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove	PartitionCount:2	ReplicationFactor:1	Configs:
	Topic: testmove	Partition: 0	Leader: 51	Replicas: 51	Isr: 51
	Topic: testmove	Partition: 1	Leader: 50	Replicas: 50	Isr: 50

 

8.  Turn off unclean.leader.election.enable and restart the cluster to ensure you have the safeguard back in place.

 

NOTE:  If you had data in the old partition on replica that wouldn't start,unless you copy that data from the old broker (as noted in step 4,) you will lose messages that were on that partition.

 

I would recommend adding more replicas per partition.  If you had a replication factor of 2 or more, you could move the replica to another server using the kafka-reassign-partitions command without having to enable unclean leader election (replication would happen automatically from the leader), but in your case, you only have 1 replica per partition, so you must enable the unclean leader election.

 

[1] https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...

Highlighted
New Contributor
Posts: 1
Registered: ‎09-16-2016

Re: Move partitions from invalid leader

thank you for your detailed reply.

 

sorry if i am digging this up, but i followed have a similar problem (topic partition 5 referencing a non-existent leader broker) and a replication factor of 1

 

i think this happened during a move of the topic from my colleagues.

 

anyhow i followed your instructions* and enabled the unclean leader election before restarting kafka as cloudera service via manager.

 

however, i am still receiving this line in the log of the supposedly new leader:

 

Broker xxx received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 1035 for partition [mytopic,5] but cannot become follower since the new leader [ID] is unavailable.

 where ID is probably an old broker (i wouldnt know) but clearly not an active broker id (of which i have 3).

 

* with the exception that i did not generate a json file but wrote it myself, since only 1 partition needed id-correction and i entered the broker that already holds the partition data on disk

Announcements