can't be moved from encryption zone

Vinitkumar — Fri, 13 Apr 2018 16:24:32 GMT

Hi,

We had logic in which computated file from hdfs path /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/_SUCCESS was moving to /bigdatahdfs/datalake/publish/prm2(external partitioned Parque table is built on top of it) , it was working fine but after recent migration to new server where encryption is enabled, its throwing series of error messages :-

[INFO] :2018-04-12 10:24:01:Wrapper:Job_name:step001_CDC: Moving Files from /bigdatahdfs/datalake/publish/prm2/merchant_location_extension to /bigdatahdfs/datalake/publish/prm2/archive/merchant_location_extension/20180405 mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/_SUCCESS can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2. mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/part-00000-m-00000.snappy.parquet can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2. mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/part-00001-m-00001.snappy.parquet can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2.

What all are steps Admin team needs to do , so that user will get privilege to move file to target HDFS directories. As a developer, I am not able to get what configuration is missing.

Re: can't be moved from encryption zone

xyao — Sat, 14 Apr 2018 06:03:07 GMT

@Vinit Pandey, for the encrypted files under HDFS encryption zone. They are only allowed to rename within the same encryption zone. You may copy to/from encryption zone which had an additional decrpt (or encrypt) overhead compared with rename.

Please refer to the document for more details:

"HDFS restricts file and directory renames across encryption zone boundaries. This includes renaming an encrypted file / directory into an unencrypted directory (e.g., hdfs dfs mv /zone/encryptedFile /home/bob), renaming an unencrypted file or directory into an encryption zone (e.g., hdfs dfs mv /home/bob/unEncryptedFile /zone), and renaming between two different encryption zones (e.g., hdfs dfs mv /home/alice/zone1/foo /home/alice/zone2). In these examples, /zone, /home/alice/zone1, and /home/alice/zone2 are encryption zones, while /home/bob is not. A rename is only allowed if the source and destination paths are in the same encryption zone, or both paths are unencrypted (not in any encryption zone)."

question Re: can't be moved from encryption zone in Archives of Support Questions (Read Only)

can't be moved from encryption zone

Re: can't be moved from encryption zone