Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

can't be moved from encryption zone

Solved Go to solution

can't be moved from encryption zone

New Contributor

Hi,

We had logic in which computated file from hdfs path /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/_SUCCESS was moving to /bigdatahdfs/datalake/publish/prm2(external partitioned Parque table is built on top of it) , it was working fine but after recent migration to new server where encryption is enabled, its throwing series of error messages :-

[INFO] :2018-04-12 10:24:01:Wrapper:Job_name:step001_CDC: Moving Files from /bigdatahdfs/datalake/publish/prm2/merchant_location_extension to /bigdatahdfs/datalake/publish/prm2/archive/merchant_location_extension/20180405 mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/_SUCCESS can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2. mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/part-00000-m-00000.snappy.parquet can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2. mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/part-00001-m-00001.snappy.parquet can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2.


What all are steps Admin team needs to do , so that user will get privilege to move file to target HDFS directories. As a developer, I am not able to get what configuration is missing.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: can't be moved from encryption zone

Rising Star

@Vinit Pandey, for the encrypted files under HDFS encryption zone. They are only allowed to rename within the same encryption zone. You may copy to/from encryption zone which had an additional decrpt (or encrypt) overhead compared with rename.

Please refer to the document for more details:

"HDFS restricts file and directory renames across encryption zone boundaries. This includes renaming an encrypted file / directory into an unencrypted directory (e.g., hdfs dfs mv /zone/encryptedFile /home/bob), renaming an unencrypted file or directory into an encryption zone (e.g., hdfs dfs mv /home/bob/unEncryptedFile /zone), and renaming between two different encryption zones (e.g., hdfs dfs mv /home/alice/zone1/foo /home/alice/zone2). In these examples, /zone, /home/alice/zone1, and /home/alice/zone2 are encryption zones, while /home/bob is not. A rename is only allowed if the source and destination paths are in the same encryption zone, or both paths are unencrypted (not in any encryption zone)."

1 REPLY 1
Highlighted

Re: can't be moved from encryption zone

Rising Star

@Vinit Pandey, for the encrypted files under HDFS encryption zone. They are only allowed to rename within the same encryption zone. You may copy to/from encryption zone which had an additional decrpt (or encrypt) overhead compared with rename.

Please refer to the document for more details:

"HDFS restricts file and directory renames across encryption zone boundaries. This includes renaming an encrypted file / directory into an unencrypted directory (e.g., hdfs dfs mv /zone/encryptedFile /home/bob), renaming an unencrypted file or directory into an encryption zone (e.g., hdfs dfs mv /home/bob/unEncryptedFile /zone), and renaming between two different encryption zones (e.g., hdfs dfs mv /home/alice/zone1/foo /home/alice/zone2). In these examples, /zone, /home/alice/zone1, and /home/alice/zone2 are encryption zones, while /home/bob is not. A rename is only allowed if the source and destination paths are in the same encryption zone, or both paths are unencrypted (not in any encryption zone)."

Don't have an account?
Coming from Hortonworks? Activate your account here