This is a partition move from one volume to another in a single broker.
I have started a kafka-reaasign-partitions of a partition from one volume to another but it never completes. Has anyone faced this ?
The directories look like the below. Here i am moving a partition from sdc to sdf volume. inodes also look fine.
desind@xxx:~#> sudo du -sh /kafka/data/sdc/prod-events-48
desind@xxx:~#> sudo du -sh /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future
desind@xxx:~#> find /kafka/data -type d -name 'prod-events-48*'
I am using kafka 2.0
I am moving a partition that is around 76G . You can see the size from my original post.
kafka-reassign-partitions --zookeeper xxxx:2181/kafka --execute --reassignment-json-file reassign.json --bootstrap-server xxxx:9092
Can you share the output of verify command for reassign partitions.
kafka-reassign-partitions --zookeeper xxxx:2181/kafka --verify --reassignment-json-file reassign.json --bootstrap-server xxxx:9092
Its been quitesome time that i ran that and dont have the session details. However i think the output would say "reassingment of replica still in progress" .
Reassignment of partition prod-events-45 completed successfully
Reassignment of replica prod-events-45-98 in progress
Reassignment of replica prod-events-45-154 completed successfully
Reassignment of replica prod-events-45-157 completed successfully
I tested this in staging cluster when there are no incoming messages, it works fine.
In production this topic events were being written at 3000 events/sec and the move did not complete. I was able to somehow stop the reassingment without causing any issues to paritions
I just retried it without the --throttle option and it works fine and i was able to sucessfully move two partitions across volumes in same broker. Summary is to use it without throttling.
I initially tried with throttle and realized it unable to complete the move as the messages are also coming in at a steady pace. i also tried increasing and removing the throttle completely and it did not complete the move. Maybe something is going on with throttle.