Support Questions
Find answers, ask questions, and share your expertise

Moving Kafka Log dirs through Ambari

Solved Go to solution
Highlighted

Moving Kafka Log dirs through Ambari

Contributor

If I want to change all the locations of log.dirs, can I just make this change in Ambari and restart kafka?

There are a couple of articles saying that data will be replicated if I just delete a folder however it implies that the migration has to be done on a per machine basis : e.g.: https://community.hortonworks.com/articles/59715/migrating-kafka-partitions-data-to-new-data-folder.....

If Ambari applies this change to all machines at once, can it still migrate the data?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Moving Kafka Log dirs through Ambari

Explorer

@Sebastian No, We have to migrate one node at a time

View solution in original post

4 REPLIES 4
Highlighted

Re: Moving Kafka Log dirs through Ambari

Explorer

@Sebastian No, We have to migrate one node at a time

View solution in original post

Re: Moving Kafka Log dirs through Ambari

Contributor

Thanks for the quick response! So a work around would be something like? (in pseudo code) :

correct_log_location = a
for each node:
  manually set log.dirs <- correct_log_location
  restart kafka
  wait till partitions have been migrated

Set correct config in Ambari and restart through Ambari
Highlighted

Re: Moving Kafka Log dirs through Ambari

Guru

Hi @Sebastian Carroll ,

why not copying over content of previous log.dirs to new ones, before restarting kafka ? this should reduce/remove the takeover of partitions

Highlighted

Re: Moving Kafka Log dirs through Ambari

Contributor

I'm not moving all directories to new places, but consolidating 8 locations to 3 - I wasn't sure how all the metadata and splits would copy over given some of the filenames are the same in each directory