Reply
Highlighted
Expert Contributor
Posts: 85
Registered: ‎05-09-2017

Reassignment of a replica across Kafka volumes in a JBOD never completes

[ Edited ]

This is a partition move from one volume to another in a single broker. 

 

I have started a kafka-reaasign-partitions of a partition from one volume to another but it never completes. Has anyone faced this ? 

  • Kafka cluster is in a healthy state 
  • Controller is fine 
  • source and destination is also fine 
  • The replica is not a leader. 
  • So far havent seen any errors in logs 

 

The directories look like the below. Here i am moving a partition from sdc to sdf volume. inodes also look fine. 


desind@xxx:~#> sudo du -sh /kafka/data/sdc/prod-events-48
76G /kafka/data/sdc/prod-events-48

 

desind@xxx:~#> sudo du -sh /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future
76G /kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future


desind@xxx:~#> find /kafka/data -type d -name 'prod-events-48*'
/kafka/data/sdf/prod-events-48.b82f63b489554ef4b2e2f4c514bc1bc0-future
/kafka/data/sdc/prod-events-48

 

 

 

Cloudera Employee
Posts: 276
Registered: ‎01-09-2014

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

Can you please provide the reassign-partitions command and files that you are using to migrate? What version of CDK are you using?

-pd
Expert Contributor
Posts: 85
Registered: ‎05-09-2017

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

[ Edited ]

I am using kafka 2.0 

 

I am moving a partition that is around 76G  . You can see the size from my original post. 

 

kafka-reassign-partitions --zookeeper xxxx:2181/kafka --execute --reassignment-json-file reassign.json --bootstrap-server xxxx:9092

New Contributor
Posts: 5
Registered: ‎08-08-2018

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

Can you share the output of verify command for reassign partitions. 

 

kafka-reassign-partitions --zookeeper xxxx:2181/kafka --verify --reassignment-json-file reassign.json --bootstrap-server xxxx:9092

Expert Contributor
Posts: 85
Registered: ‎05-09-2017

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

Its been quitesome time that i ran that and dont have the session details. However i think the output would say "reassingment of replica still in progress" .

 

Reassignment of partition prod-events-45 completed successfully
Reassignment of replica prod-events-45-98 in progress
Reassignment of replica prod-events-45-154 completed successfully
Reassignment of replica prod-events-45-157 completed successfully

 

I tested this in staging cluster when there are no incoming messages, it works fine.

In production this topic events were being written at 3000 events/sec and the move did not complete. I was able to somehow stop the reassingment without causing any issues to paritions 

Expert Contributor
Posts: 85
Registered: ‎05-09-2017

Re: Reassignment of a replica across Kafka volumes in a JBOD never completes

[ Edited ]

I just retried it without the --throttle option and it works fine and i was able to sucessfully move two partitions across volumes in same broker. Summary is to use it without throttling. 

 

I initially tried with throttle and realized it unable to complete the move as the messages are also coming in at a steady pace. i also tried increasing and removing the throttle completely and it did not complete the move. Maybe something is going on with throttle.  

Announcements
New solutions