Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Move partitions from invalid leader

Move partitions from invalid leader

Rising Star

Hi ,

I have a situation .

i have 3 broker(id) ==> 58,57,56,

some how broker 57 went invalid 

 

hduser@node01:~$ kafka-topics --zookeeper node01:2181,node02:2181,rac1:2181 --describe

Topic:clouder PartitionCount:1 ReplicationFactor:2 Configs:
Topic: clouder Partition: 0 Leader: 56 Replicas: 56,57 Isr: 56
Topic:truckevent PartitionCount:2 ReplicationFactor:1 Configs:
Topic: truckevent Partition: 0 Leader: -1 Replicas: 57 Isr: 57
Topic: truckevent Partition: 1 Leader: 56 Replicas: 56 Isr: 56

 

 i moved to replica from 57 to 58 

 

root@node01:/opt/TruckEvents# kafka-topics --zookeeper node01:2181,node02:2181,rac1:2181 --describe
Topic:clouder PartitionCount:1 ReplicationFactor:2 Configs:
Topic: clouder Partition: 0 Leader: 56 Replicas: 56,57 Isr: 56
Topic:truckevent PartitionCount:2 ReplicationFactor:2 Configs:
Topic: truckevent Partition: 0 Leader: -1 Replicas: 58,57 Isr: 57
Topic: truckevent Partition: 1 Leader: 56 Replicas: 56 Isr: 56

 

but i also want to move leadership to 58 and tried 

 

root@node01:/opt/TruckEvents# kafka-preferred-replica-election --zookeeper node01:2181,node02:2181,rac1:2181
Successfully started preferred replica election for partitions Set([truckevent,1], [truckevent,0], [clouder,0])

nothing changed and Leader is still 57 .

 

how can we change the leader is its invaid ?

 

 

3 REPLIES 3

Re: Move partitions from invalid leader

Rising Star
Anyone ? pls help

Re: Move partitions from invalid leader

Super Collaborator

For this partition:

Topic: truckevent Partition: 0 Leader: -1 Replicas: 57 Isr: 57

 

Since you only have replication-factor of 1, you don't have another replica to bring up as the leader.  In this instance, you can move the replica to a running broker, but you will have to enable unclean leader election, as that new replica isn't in the ISR for partition 0.  You won't have any of the data that is in that partition, as it all resides on broker 57.  If you can move that data to broker 58, you would preserve the messages in that partition.

 

If you need to go that route, you would need to use the kafka-reassign-partitions command to take the single partition (truckevent partition 0 on broker 57) and move it to another broker. See the full documentation here [1]

 

 

In this example, we are moving partition 0 for the testmove topic from broker 52 to 51 in this example:

 

 

 

[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove    PartitionCount:2    ReplicationFactor:2    Configs:
    Topic: testmove    Partition: 0    Leader: -1    Replicas: 52    Isr: 52
    Topic: testmove    Partition: 1    Leader: 50    Replicas: 50    Isr: 50

 

 

1. Create the topictomove.json file:

 

echo '{"topics": [{"topic": "testmove"}], "version":1 }' > topictomove.json

 

 

2. generate the partitions to reassign using the --generate option:

 

root@host-1 ~]# kafka-reassign-partitions --zookeeper ${HOSTNAME}:2181 --generate --broker-list 51,50 --topics-to-move-json-file topictomove.json
Current partition replica assignment

{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[52]},{"topic":"testmove","partition":1,"replicas":[50]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[50]},{"topic":"testmove","partition":1,"replicas":[51]}]}

 

 

3.  Take the output from the "Proposed partition reassignment configuration" and include just the partition you want to move, in a file called reassign.json.  In this instance, we are moving partition0 from replica 52 to replica 51 so the broker id listed in the json file will be the new target:

 

[root@host-1 ~]# cat reassign.json
{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[51]}]}

 

4.  If possible, copy the partition folder from the invalid broker to the new broker.  This will ensure the partition contains all the data on the new broker:

 

rsync -ave ssh /var/local/kafka/data/testmove-0 broker51:/var/local/kafka/data/

 

 

5. Use the --execute command to make the changes:

 

[root@host-1 ~]# kafka-reassign-partitions --zookeeper ${HOSTNAME}:2181 --execute --reassignment-json-file reassign.json
Current partition replica assignment

{"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[52]},{"topic":"testmove","partition":1,"replicas":[50]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"testmove","partition":0,"replicas":[51]}]}

 

6.  At this time, the new replica will show up, but its not in the ISR list, and it can't be elected leader:

 

[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove    PartitionCount:2    ReplicationFactor:2    Configs:
    Topic: testmove    Partition: 0    Leader: -1    Replicas: 51,52    Isr: 52
    Topic: testmove    Partition: 1    Leader: 50    Replicas: 50    Isr: 50

And in the logs on the new broker, you will see the following:

 

 

2016-08-05 10:45:17,119 ERROR state.change.logger: Broker 51 received LeaderAndIsrRequest with correlation id 126 from controller 50 epoch 4 for partition [testmove,0] but cannot become follower since the new leader -1 is unavailable.

 

7.  You will have to enable unclean.leader.election.enable in Cloudera Manager for the Kafka service configuration, and restart the kafka service.  The broker that is the active controller needs to be restarted with this flag enabled to properly allow the new leader to be elected, even though it is not in the ISR list.

 

7.  The new replica is now shown as the new leader:

 

[root@host-1 ~]# kafka-topics --describe --topic testmove --zookeeper ${HOSTNAME}:2181
Topic:testmove	PartitionCount:2	ReplicationFactor:1	Configs:
	Topic: testmove	Partition: 0	Leader: 51	Replicas: 51	Isr: 51
	Topic: testmove	Partition: 1	Leader: 50	Replicas: 50	Isr: 50

 

8.  Turn off unclean.leader.election.enable and restart the cluster to ensure you have the safeguard back in place.

 

NOTE:  If you had data in the old partition on replica that wouldn't start,unless you copy that data from the old broker (as noted in step 4,) you will lose messages that were on that partition.

 

I would recommend adding more replicas per partition.  If you had a replication factor of 2 or more, you could move the replica to another server using the kafka-reassign-partitions command without having to enable unclean leader election (replication would happen automatically from the leader), but in your case, you only have 1 replica per partition, so you must enable the unclean leader election.

 

[1] https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignParti...

Re: Move partitions from invalid leader

New Contributor

thank you for your detailed reply.

 

sorry if i am digging this up, but i followed have a similar problem (topic partition 5 referencing a non-existent leader broker) and a replication factor of 1

 

i think this happened during a move of the topic from my colleagues.

 

anyhow i followed your instructions* and enabled the unclean leader election before restarting kafka as cloudera service via manager.

 

however, i am still receiving this line in the log of the supposedly new leader:

 

Broker xxx received LeaderAndIsrRequest with correlation id 1 from controller 1 epoch 1035 for partition [mytopic,5] but cannot become follower since the new leader [ID] is unavailable.

 where ID is probably an old broker (i wouldnt know) but clearly not an active broker id (of which i have 3).

 

* with the exception that i did not generate a json file but wrote it myself, since only 1 partition needed id-correction and i entered the broker that already holds the partition data on disk