can't be moved from encryption zone

[#1]

Hi,

We had logic in which computated file from hdfs path /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/_SUCCESS was moving to /bigdatahdfs/datalake/publish/prm2(external partitioned Parque table is built on top of it) , it was working fine but after recent migration to new server where encryption is enabled, its throwing series of error messages :-

[INFO] :2018-04-12 10:24:01:Wrapper:Job_name:step001_CDC: Moving Files from /bigdatahdfs/datalake/publish/prm2/merchant_location_extension to /bigdatahdfs/datalake/publish/prm2/archive/merchant_location_extension/20180405 mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/_SUCCESS can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2. mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/part-00000-m-00000.snappy.parquet can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2. mv: /bigdatahdfs/datalake/raw/prm2/temp/merchant_location_extension/part-00001-m-00001.snappy.parquet can't be moved from encryption zone /bigdatahdfs/datalake/raw/prm2 to encryption zone /bigdatahdfs/datalake/publish/prm2.

What all are steps Admin team needs to do , so that user will get privilege to move file to target HDFS directories. As a developer, I am not able to get what configuration is missing.

[#2 · Accepted solution]

@Vinit Pandey, for the encrypted files under HDFS encryption zone. They are only allowed to rename within the same encryption zone. You may copy to/from encryption zone which had an additional decrpt (or encrypt) overhead compared with rename.

Please refer to the document for more details:

"HDFS restricts file and directory renames across encryption zone boundaries. This includes renaming an encrypted file / directory into an unencrypted directory (e.g., hdfs dfs mv /zone/encryptedFile /home/bob), renaming an unencrypted file or directory into an encryption zone (e.g., hdfs dfs mv /home/bob/unEncryptedFile /zone), and renaming between two different encryption zones (e.g., hdfs dfs mv /home/alice/zone1/foo /home/alice/zone2). In these examples, /zone, /home/alice/zone1, and /home/alice/zone2 are encryption zones, while /home/bob is not. A rename is only allowed if the source and destination paths are in the same encryption zone, or both paths are unencrypted (not in any encryption zone)."