Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Explorer

This is an unsupported technology and a concept which hasn't been explored yet.

There's no real modification time concept in object stores. It has just creation time, which is that of the observed time at the far end. If you upload a file to a remote timezone, you may get that as your time.

The underlying issue here is not a bug. It is just a feature that distcp -update relies on using file checksums for comparing HDFS files, and (a) not all stores export their checksum through the Hadoop API (WASB does, s3a doesn't yet).

In addition, because the checksums are different between blobstores and HDFS, you can't use checksum difference as a cue for files being changed.

Note that this also occurs when trying to copy between HDFS encryption zones, as the checksums of the encrypted files will differ.

141 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎09-29-2017 08:01 AM
Updated by:
 
Contributors
Top Kudoed Authors