Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Moving Kafka Log dirs through Ambari

Solved Go to solution

Moving Kafka Log dirs through Ambari

Contributor

If I want to change all the locations of log.dirs, can I just make this change in Ambari and restart kafka?

There are a couple of articles saying that data will be replicated if I just delete a folder however it implies that the migration has to be done on a per machine basis : e.g.: https://community.hortonworks.com/articles/59715/migrating-kafka-partitions-data-to-new-data-folder.....

If Ambari applies this change to all machines at once, can it still migrate the data?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Moving Kafka Log dirs through Ambari

New Contributor

@Sebastian No, We have to migrate one node at a time

4 REPLIES 4

Re: Moving Kafka Log dirs through Ambari

New Contributor

@Sebastian No, We have to migrate one node at a time

Highlighted

Re: Moving Kafka Log dirs through Ambari

Contributor

Thanks for the quick response! So a work around would be something like? (in pseudo code) :

correct_log_location = a
for each node:
  manually set log.dirs <- correct_log_location
  restart kafka
  wait till partitions have been migrated

Set correct config in Ambari and restart through Ambari

Re: Moving Kafka Log dirs through Ambari

Guru

Hi @Sebastian Carroll ,

why not copying over content of previous log.dirs to new ones, before restarting kafka ? this should reduce/remove the takeover of partitions

Re: Moving Kafka Log dirs through Ambari

Contributor

I'm not moving all directories to new places, but consolidating 8 locations to 3 - I wasn't sure how all the metadata and splits would copy over given some of the filenames are the same in each directory

Don't have an account?
Coming from Hortonworks? Activate your account here