Is there a way to delete an HDFS directory which might have a snapshottable directory?

Question

There are user directories under hdfs:///user/

When a user `foo` retires, I need to delete a root directory for the user: hdfs:///user/foo

However, it occasionally fails because of the snapshottable directories under the user root directory.

Checking all the subdirectories whether it's snapshottable, disallowing snapshottable subdirectories , and then deleting the user's root directory doesn't seem to be the best way.

(Or if this is the best way, I cannot come up with a simple code..)

Is there a command to disallow snapshots for all the subdirectories?

How can I effectively delete a directory that might have snapshots?

Shelton · Accepted Answer

@sipocootap2

Here is a walkthrough on how to delete a snapshot

Created a directory

$ hdfs dfs -mkdir -p /app/tomtest

Changed the owner

$ hdfs dfs -chown -R tom:developer /app/tomtest

To be able to create a snapshot the directory has to be snapshottable

$ hdfs dfsadmin -allowSnapshot /app/tomtest
Allowing snaphot on /app/tomtest succeeded

Now I created 3 snapshots

$ hdfs dfs -createSnapshot /app/tomtest sipo
Created snapshot /app/tomtest/.snapshot/sipo

$ hdfs dfs -createSnapshot /app/tomtest coo
Created snapshot /app/tomtest/.snapshot/coo

$ hdfs dfs -createSnapshot /app/tomtest tap2
Created snapshot /app/tomtest/.snapshot/tap2

Confirm the directory is snapshottable

$ hdfs lsSnapshottableDir
drwxr-xr-x 0 tom developer 0 2021-07-26 23:14 3 65536 /app/tomtest

List all the snapshots in the directory

$ hdfs dfs -ls /app/tomtest/.snapshot
Found 3 items
drwxr-xr-x - tom developer 0 2021-07-26 23:14 /app/tomtest/.snapshot/coo
drwxr-xr-x - tom developer 0 2021-07-26 23:14 /app/tomtest/.snapshot/sipo
drwxr-xr-x - tom developer 0 2021-07-26 23:14 /app/tomtest/.snapshot/tap2

Now I need to delete the snapshot coo

$ hdfs dfs -deleteSnapshot /app/tomtest/ coo

Confirm the snapshot is gone

$ hdfs dfs -ls /app/tomtest/.snapshot
Found 2 items
drwxr-xr-x - tom developer 0 2021-07-26 23:14 /app/tomtest/.snapshot/sipo
drwxr-xr-x - tom developer 0 2021-07-26 23:14 /app/tomtest/.snapshot/tap2

Voila

To delete a snapshot the format is hdfs dfs -deleteSnapshot <path> <snapshotName> i.e

hdfs dfs -deleteSnapshot /app/tomtest/ coo notice the space and omittion of the .snapshot as all .(dot) files the snapshot directory is not visible with normal hdfs command

The -ls command gives 0 results

$ hdfs dfs -ls /app/tomtest/

The special command shows the 2 remaining snapshots

$ hdfs dfs -ls /app/tomtest/.snapshot
Found 2 items
drwxr-xr-x - tom developer 0 2021-07-26 23:14 /app/tomtest/.snapshot/sipo
drwxr-xr-x - tom developer 0 2021-07-26 23:14 /app/tomtest/.snapshot/tap2

Is there a command to disallow snapshots for all the subdirectories? Yes there is only after you have deleted all the snapshots therein demo, or better at directory creation time you can disallow snapshots

$ hdfs dfsadmin -disallowSnapshot /app/tomtest/
disallowSnapshot: The directory /app/tomtest has snapshot(s). Please redo the operation after removing all the snapshots.

The only way I have found which works when for me and permits me to have a cup of coffee is to first list all the snapshots and copy-paste the delete even if there are 60 snapshots it works and I only get back when the snapshots are gone or better still do something else while the deletion is going on not automated though the example

The below would run concurrently

hdfs dfs -deleteSnapshot /app/tomtest/ sipo
.....
....
hdfs dfs -deleteSnapshot /app/tomtest/ tap2

-deleteSnapshot skips trash by default!

Happy hadooping

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.