question Re: Reassignment of a replica across Kafka volumes in a JBOD never completes in Support Questions

Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Fri, 16 Sep 2022 14:24:24 GMT

This is a partition move from one volume to another in a single broker.

I have started a kafka-reaasign-partitions of a partition from one volume to another but it never completes. Has anyone faced this ?

Kafka cluster is in a healthy state
Controller is fine
source and destination is also fine
The replica is not a leader.
So far havent seen any errors in logs

The directories look like the below. Here i am moving a partition from sdc to sdf volume. inodes also look fine.

desind@xxx:~#> sudo du -sh /kafka/data/sdc/prod-events-48
76G /kafka/data/sdc/prod-events-48

desind@xxx:~#> sudo du -sh /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future
76G /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future

desind@xxx:~#> find /kafka/data -type d -name 'prod-events-48*'
/kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future
/kafka/data/sdc/prod-events-48

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

pdvorak — Wed, 05 Jun 2019 15:35:35 GMT

Can you please provide the reassign-partitions command and files that you are using to migrate? What version of CDK are you using?

-pd

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Wed, 05 Jun 2019 16:59:14 GMT

I am using kafka 2.0

I am moving a partition that is around 76G . You can see the size from my original post.

kafka-reassign-partitions --zookeeper xxxx:2181/kafka --execute --reassignment-json-file reassign.json --bootstrap-server xxxx:9092

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

Rohit_Pathak — Wed, 05 Jun 2019 23:27:48 GMT

Can you share the output of verify command for reassign partitions.

kafka-reassign-partitions --zookeeper xxxx:2181/kafka --verify --reassignment-json-file reassign.json --bootstrap-server xxxx:9092

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Thu, 06 Jun 2019 00:52:35 GMT

Its been quitesome time that i ran that and dont have the session details. However i think the output would say "reassingment of replica still in progress" .

Reassignment of partition prod-events-45 completed successfully
Reassignment of replica prod-events-45-98 in progress
Reassignment of replica prod-events-45-154 completed successfully
Reassignment of replica prod-events-45-157 completed successfully

I tested this in staging cluster when there are no incoming messages, it works fine.

In production this topic events were being written at 3000 events/sec and the move did not complete. I was able to somehow stop the reassingment without causing any issues to paritions

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Mon, 17 Jun 2019 19:04:52 GMT

I just retried it without the --throttle option and it works fine and i was able to sucessfully move two partitions across volumes in same broker. Summary is to use it without throttling.

I initially tried with throttle and realized it unable to complete the move as the messages are also coming in at a steady pace. i also tried increasing and removing the throttle completely and it did not complete the move. Maybe something is going on with throttle.

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

VijayReddy — Wed, 27 Nov 2019 02:07:13 GMT

Can you please let me know whether the issue resolved?Am also facing the same issue.Reassignment is still running since 2 days. I ran the reassignment without throttling.

Also after this am seeing old logs files still exists and retention.ms and retention.bytes is not working

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Wed, 27 Nov 2019 02:17:53 GMT

Do you have the original assignment ?

You can stop it by passing the original assignment. It will just revert back. I did this couple of times and it works.

Now regarding why its not completing can be an IO issue.
Did you configure this value in the cluster num.replica.alter.log.dirs.threads. ? If not you might have to set it.
Also might have to set intra.broker.throttled.rate using the safety valve.

I read it in this KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-113%3A+Support+replicas+movement+between+log+directories

Test in staging and then you can implement in prod.

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

VijayReddy — Wed, 27 Nov 2019 02:35:02 GMT

Before Reassignment the replicas are as below for Topic XXX

Topic: XXXX Partition: 1 Leader: 3 Replicas: 3,0,1

Original Reassignment json file:

{"version":1,"partitions":[{"topic":"XXXXX","partition":1,"replicas":[3,0,8],"log_dirs":["/data1/kafka","/data3/kafka","/data1/kafka"]}]}

For 3 node am moving the partition 1 data from data2 to data1 and for 0 node am moving the partition 1 data from data1 to data3 and for node 8 it is new node

Status of Verify as below

Status of partition reassignment:
Reassignment of partition XXXX-1 completed successfully
Reassignment of replica XXXX-1-3 is still in progress
Reassignment of replica XXXX-1-0 is still in progress
Reassignment of replica XXXX-1-8 completed successfully

I didnt set any values which you mentioned. The problem now here is old logs are not getting deleted and in the logs as seeing error as below

 INFO [Partition XXXX-1 broker=3] XXXX-1 starts at Leader Epoch 101 from offset 10647332982. Previous Leader Epoch was: 100 (kafka.cluster.Partition)
INFO [Partition XXXX-tapsampleapp-1 broker=3] XXXX-tapsampleapp-1 starts at Leader Epoch 99 from offset 555229. Previous Leader Epoch was: 98 (kafka.cluster.Partition)
INFO [ReplicaAlterLogDirsThread-1]: Partition XXXX-1 has an older epoch (100) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaAlterLogDirsThread)

Intially it created one future folder and tried to do reasignment and then stopped suddenly.

Now my main concern is old logs are not getting deleted and file system is getting full

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Wed, 27 Nov 2019 02:45:02 GMT

To stop the move run the reassignment again with --execute option using the original

{"version":1,"partitions":[{"topic":"XXXXX","partition":1,"replicas":[3,0,8],"log_dirs":["/data1/kafka","/data3/kafka","/data1/kafka"]}]}

Then what you can do is instead of moving it on the same broker try to move the partition to a different broker

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

VijayReddy — Wed, 27 Nov 2019 02:54:19 GMT

I ran the reassignment previously using the below command with the json which i shared earlier. Do you want to run the same once again?

/kafka-reassign-partitions.sh --execute --bootstrap-server XXXX:9092 --reassignment-json-file ./test.json --zookeeper XXXXX:2181

As this is production i want to confirm once again

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Wed, 27 Nov 2019 03:06:04 GMT

Yes. What we are doing here is that we are stopping the reassignment that is not progressing. And to do that you run it with the original json file and it will stop and revert back.

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

VijayReddy — Wed, 27 Nov 2019 16:16:02 GMT

Hi, i was able to replicate the same issue in Test environment. I did rerun the --execute command for the same json file but still the same.Nothing changed after re-executing the command once again.

Any other suggestions?

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Wed, 27 Nov 2019 16:22:59 GMT

Thats wierd.

So in the log.dirs specify any,any,any and see if that works

Example

{
"partitions": [{
"topic": "foo",
"partition": 1,
"replicas": [1, 2, 3],
"log_dirs": ["any", "any", "any"]
}],
"version": 1
}

This is what documentation says "Broker will cancel existing movement of the replica if "any" is specified as destination log directory."

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

VijayReddy — Wed, 27 Nov 2019 16:46:28 GMT

With ANY it worked. And also i need to restart the kafka service to delete the old segment logs.

Thanks very much for your support

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

VijayReddy — Wed, 27 Nov 2019 17:23:27 GMT

I implemented the same thing in Production and reassignment is showing as completed. But old segmented logs and future folder is not getting deleted.

From the logs i can see as below

INFO [ReplicaAlterLogDirsThread-1]: Partition XXXX-1 has an older epoch (102) than the current leader. Will await the new LeaderAndIsr state before resuming fetching. (kafka.server.ReplicaAlterLogDirsThread)
INFO [Partition XXX-1 broker=3] XXX-1 starts at Leader Epoch 106 from offset 0. Previous Leader Epoch was: 105 (kafka.cluster.Partition)

After restarting the service, it tried to copy the data from one folder to other folder for some time and then stopped

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

desind — Wed, 27 Nov 2019 19:15:08 GMT

Now that the move has stopped/completed.

Verify if there are no under-replicated OR offline partitions and also verify if there are 3 replicas and all are in sync (depends on your replication factor)

You can delete those partitions manually and just restart the broker.

If replicas are out of sync they must come back in sync if unclean.leader.election is true

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

VijayReddy — Wed, 27 Nov 2019 19:42:19 GMT

Now all is fine. After deleting the folders and restarting the service the issue is resolved.

Thanks very much for the support

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

WitchDocotor — Fri, 24 Apr 2020 01:49:31 GMT

we had the some issue, and I tried this, it seems not work.