@bigdata.neophyte Answers inline.
How does snapshots help for Disaster Recovery? What are the best practices around using snapshots for DR purposes? Especially trying to understand when data is directly stored on HDFS, Hive data and HBase data Snapshot will not be best option for DR. Can a directory be deleted using hdfs dfs -rmr -skipTrash /data/snapshot-dir? Or is it that all the snapshots have to be deleted first and then the snapshotting be disabled before allowing the directory be deleted? No. If the directory is snapshottable it cannot be deleted. As I understand, no data is copied for snapshots, but only metadata is maintained for the blocks added/ modified / deleted. If that’s the case, just wondering what happens when the comamnd hdfs dfs -rm /data/snapshot-dir/file1 is run. Will the file be moved to the trash? If so, will the snapshot maintain the reference to the entry in trash? Will trach eviction has any impact in this case? If a file is deleted from snapshottable directory it will be stored in .snapshot folder and you can copy data back. For eg. [hdfs@node1 ~]$ hdfs dfs -rm -r -skipTrash /tmp/test/anaconda-ks.cfg
Deleted /tmp/test/anaconda-ks.cfg
[hdfs@node1 ~]$ hadoop fs -ls /tmp/test/.snapshot/s20160526-022510.203
-rw-r--r-- 3 hdfs hdfs 1155 2016-05-26 02:23 /tmp/test/.snapshot/s20160526-022510.203/anaconda-ks.cfg
[hdfs@node1 ~]$ hadoop fs -cp /tmp/test/.snapshot/s20160526-022510.203/anaconda-ks.cfg /tmp/test You will be able to see the data back in the file. What happens when one of the sub-directory under the snapshot directory is deleted? For example, if the command hdfs dfs -rmr -skipTrash /data/sub-dir is run? Can the data be recovered from snapshots? If the subdirectory already exist when "
hdfs dfsadmin -allowSnapshot" was run then if we delete directory it will be there in ".snapshot" folder. Eg "
/user/test1/.snapshot/s20160526-025323.341/subdir/ambari.properties.2" If the subdirectory is created later then it will not be backed up. Can snapshots be deleted / archived automatically based on policies, for example time-based? In the above example, how long will the sub-dir data be maintained in the snapshot? You can create custom script to delete/archive snap based on policies. The snapshot will be maintained till we delete it. How does snapshots work along with HDFS quotas. For example, assume a directory with a quota of 1GB with snapshotting enabled. Assume the directory is closer to its full quota and a user deleted a large file to store some other dataset. Will the new data be allowed to be saved to the directory or will the operation be stopped because the quota limits have been exceeded? It will will allow you to exceed the quota mentioned. It will give warning. Apologies if some of the questions doesn’t make sense. I am still trying to understand these concepts at a ground level.
... View more
