Moving Kafka Log dirs through Ambari

[#1]

If I want to change all the locations of log.dirs, can I just make this change in Ambari and restart kafka?

There are a couple of articles saying that data will be replicated if I just delete a folder however it implies that the migration has to be done on a per machine basis : e.g.: https://community.hortonworks.com/articles/59715/migrating-kafka-partitions-data-to-new-data-folder.....

If Ambari applies this change to all machines at once, can it still migrate the data?

[#2 · Accepted solution]

@Sebastian No, We have to migrate one node at a time

[#3]

Thanks for the quick response! So a work around would be something like? (in pseudo code) :

correct_log_location = a
for each node:
  manually set log.dirs <- correct_log_location
  restart kafka
  wait till partitions have been migrated

Set correct config in Ambari and restart through Ambari

[#4]

Hi @Sebastian Carroll ,

why not copying over content of previous log.dirs to new ones, before restarting kafka ? this should reduce/remove the takeover of partitions

[#5]

I'm not moving all directories to new places, but consolidating 8 locations to 3 - I wasn't sure how all the metadata and splits would copy over given some of the filenames are the same in each directory