Reply
Highlighted
Expert Contributor
Posts: 256
Registered: ‎01-25-2017

distcp snapshot managemnt

Hello All,

 

I'm using distcp with snapshot and going to implement the snapshot restore in the distcp in my next CDH upgrade to 5.10.

 

Want to share with you how i'm thinking to manage the snapshot cycle and want to gain from your knowledge if i can manage it in a better way, and sure there is alot of edge cases that the cycle will corruped, will be happy to get your feedback and any new suggestion is much appreciated, the starting point where both source and destination has identical s0 snapshot.

 

 

#!/bin/bash -x


hdfs dfs -createSnapshot /fawzesource s1
hadoop distcp -diff s0 s1 /fawzesource /fawzedestination
if [ $? -eq 0 ]
then

hdfs dfs -createSnapshot /fawzedestination s1
hdfs dfs -deleteSnapshot /fawzesource s0
hdfs dfs -renameSnapshot //fawzesource s1 s0
hdfs dfs -deleteSnapshot //fawzedestination s0
hdfs dfs -renameSnapshot //fawzedestination s1 s0
else
hdfs dfs -deleteSnapshot //fawzedestination s2
hdfs dfs -createSnapshot //fawzedestination s2
hadoop distcp -rdiff s2 s0 //fawzesource //fawzedestination
if [ $? -eq 0 ]
then
hdfs dfs -deleteSnapshot //fawzedestination s2
hdfs dfs -deleteSnapshot //fawzedestination s0
hdfs dfs -createSnapshot //fawzedestination s0


fi


fi

Expert Contributor
Posts: 256
Registered: ‎01-25-2017

Re: distcp snapshot managemnt

Any suggestion sor thoughts?

Announcements