Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Understanding Distcp -delete option removing directories

Understanding Distcp -delete option removing directories

I am trying to understand distcp delete option basically each time I do distcp i would like to overwrite destination directories.

The overwrite option only does it with files so if there is a file in destination with same content of a file in source but a different name would not be overriden but I would like to do it at the directory level as well.

From the hadoop docs

Delete the files existing in the dst but not in src

https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html

Thank you

1 REPLY 1
Highlighted

Re: Understanding Distcp -delete option removing directories

@na

I think thats expected behaaviour. For your scenario I would better suggest to go for DistCp between Snapshot Difference.

distcp -update -diff -delete /source /destination

How to Use This Feature

To use this feature, you should first make sure all assumptions are met. Typical steps are described as follows:

  1. Create snapshot s0 in the source directory.
  2. Issue a default distcp command that copies everything from s0 to the target directory (command line is like distcp -update <sourceDir>/.snapshot/s0 <targetDir>).
  3. Create snapshot s0 in the target dir.
  4. Make some changes in the source dir.
  5. Create a new snapshot s1, and issue a distcp command like distcp -update -diff s0 s1 <sourceDir> <targetDir> to copy all changes between s0 and s1 to the target directory.
  6. Create a snapshot with the same name s1 in the target dir.
  7. Repeat steps 4 to 6 with a new snapshot name—for example, s2.
Link

Hope this helps you.

Don't have an account?
Coming from Hortonworks? Activate your account here