question Re: Moving Kafka Log dirs through Ambari in Archives of Support Questions (Read Only)

Moving Kafka Log dirs through Ambari

scarroll — Thu, 11 May 2017 16:44:50 GMT

If I want to change all the locations of log.dirs, can I just make this change in Ambari and restart kafka?

There are a couple of articles saying that data will be replicated if I just delete a folder however it implies that the migration has to be done on a per machine basis : e.g.: https://community.hortonworks.com/articles/59715/migrating-kafka-partitions-data-to-new-data-folder.html.

If Ambari applies this change to all machines at once, can it still migrate the data?

Re: Moving Kafka Log dirs through Ambari

mkumar2 — Thu, 11 May 2017 17:59:50 GMT

@Sebastian No, We have to migrate one node at a time

Re: Moving Kafka Log dirs through Ambari

scarroll — Thu, 11 May 2017 18:05:41 GMT

Thanks for the quick response! So a work around would be something like? (in pseudo code) :

correct_log_location = a
for each node:
  manually set log.dirs <- correct_log_location
  restart kafka
  wait till partitions have been migrated

Set correct config in Ambari and restart through Ambari

Re: Moving Kafka Log dirs through Ambari

geko — Fri, 12 May 2017 01:48:22 GMT

Hi @Sebastian Carroll ,

why not copying over content of previous log.dirs to new ones, before restarting kafka ? this should reduce/remove the takeover of partitions

Re: Moving Kafka Log dirs through Ambari

scarroll — Mon, 15 May 2017 22:07:04 GMT

I'm not moving all directories to new places, but consolidating 8 locations to 3 - I wasn't sure how all the metadata and splits would copy over given some of the filenames are the same in each directory