question Re: Is it possible to use trash in HDFS encryption zone in Support Questions

Is it possible to use trash in HDFS encryption zone?

naveen1 — Fri, 16 Sep 2022 09:59:18 GMT

I am using -skipTrash to delete an HDFS file from encryption zone. Is there any way that I can use trash to recover a deleted file from encryption zone?

Re: Is it possible to use trash in HDFS encryption zone

Harsh J — Mon, 25 Jan 2016 09:43:52 GMT

Regardless of an encryption zone in use or not, an 'hadoop fs -rm' with
-skipTrash will permanently remove a file unless you have a snapshot
referencing it. If you want to use the trash ability, you need to use
'hadoop fs -rm' without -skipTrash.

Encryption zones merely create the blocks with encrypted data and associate
keys with it. Other HDFS behaviour remains the same, with the exception
being that you cannot move a file within one EZ to another, or move it
outside of the EZ.

Re: Is it possible to use trash in HDFS encryption zone

naveen1 — Mon, 25 Jan 2016 14:01:03 GMT

I didnt mean to use skipTrash, I was suggested to use that as I couldnt delete a file from encryption zone. If there is any way to use trash for encryption zone please let me know.

Re: Is it possible to use trash in HDFS encryption zone

SiddeshSamarth — Mon, 25 Jan 2016 16:37:02 GMT

I had the same question too.

When we try to delete something from encryption zone, it says the directory cannot be moved to /user/xyz/.trash.

So we have to forcefully use -skipTrash option.

So, is there a way to delete a file from encyption zone without using -skipTrash option?

Re: Is it possible to use trash in HDFS encryption zone

Consult — Wed, 27 Jan 2016 16:27:40 GMT

Hello naveen1,

For a file in encrypted zone a LOGICAL workaround could be to introduce .trash folder in same zone. This folder can be used as destination to move the redundant file(s) before removing them permanently.

In a scenario where full encrypted zone is deleted, MAY use trash bin if trash feature is enable (Please test before you implement. I haven't tested it yet.)

Hope that helps.

Re: Is it possible to use trash in HDFS encryption zone

Consult — Wed, 27 Jan 2016 16:29:30 GMT

Hello SiddeshSamarth,

Currently, there is no option to send encrypted files to global bin for obvious reasons.

Re: Is it possible to use trash in HDFS encryption zone

SiddeshSamarth — Wed, 27 Jan 2016 16:44:52 GMT

So, Do I have to forcibly use –skipTrash to delete anything in Encrypted zone or is there any workaround?

Re: Is it possible to use trash in HDFS encryption zone

Consult — Thu, 28 Jan 2016 08:11:13 GMT

Hello SiddeshSamarth,

Re: Is it possible to use trash in HDFS encryption zone

Harsh J — Thu, 28 Jan 2016 08:34:44 GMT

Consult has already posted some alternative solutions, but its worth understanding why it fails.

The effective issue is this:

When you use 'fs -rm' with trash enabled, we move the file to the authenticated user's /user/{user.name}/.Trash sub-directory. For ex., if the path of deletion is '/data/myapp/part-00000.gz', and the user you delete it as is 'hive', then the trash feature moves it to directory '/user/hive/.Trash/Current/'.

When encrypted zones come into play, HDFS disallows you from moving a file from one Encrypted Zone to another Encrypted Zone, as well as from within an Encrypted Zone to a non-Encrypted Zone. This is for security reasons, and ties into how the encryption zone features of HDFS are managed globally within a directory (zone), vs. arbitrary files holding all of the necessary info independently.

So if /data/ is an EZ, but /user/hive is not, or is a separate EZ, then the trash moving will fail expectedly.

But if / is the EZ, then the moves may work, since both paths come under it.

What Consult proposes is a manual step (i.e. use hadoop fs -mv instead of hadoop fs -rm), and keep a manually created /data/.Trash directory to move the files into, followed by scripts to periodically clean it (i.e. Bring-Your-Own-Trash). Its not a great solution but its what may work if you need some data retention.

Another option is to consider using limited and periodic snapshots (via BDR, etc.), which give you similar (but not exactly the same) data retention capabilities.

Re: Is it possible to use trash in HDFS encryption zone

naveen1 — Fri, 29 Jan 2016 01:39:49 GMT

Thank you all for your time, logical workaround sounds good to me.