Expert Contributor
Posts: 252
Registered: ‎01-25-2017

Re: Distcp copy start to fail after the upgrade to CDH5.10

So just to make things more clear and useful for who is going to use this feature or using the distcp diff in his current CDH version.


1- If you are using the snapshot diff in your current version (prior to CDH5.10 or CDH5.9.1), the distcp was able to overcome the distcp failure by listing all the source dir and run the disctp from scratch), in the new versions distcp will not overcomes such issue when distcp fail or interrupted during the run, and will fail all time in the next runs.


2- To overcome this you have to use the snapshot restore and restore the your destination hdfs folder to the state before the distcp failure.


3- The distcp snapshot command should be like this: hadoop disctp -rdiff s1 s0 source_folder destination_folder, and here the s1 is a snapshot at the destination and newer than s0, what will happen after the success of the distcp -rdiff that the destination will be restore to s0 which is the state before the distcp failure.


4- The most challenging thing will be how to manage the snaphot cycle during the distcp diff and distcp rdiff.


example how i'm doing this and working to enhance it




#!/bin/bash -x

hdfs dfs -createSnapshot /fawzesource s1
hadoop distcp -diff s0 s1 /fawzesource /fawzedestination
if [ $? -eq 0 ]

hdfs dfs -createSnapshot /fawzedestination s1
hdfs dfs -deleteSnapshot /fawzesource s0
hdfs dfs -renameSnapshot //fawzesource s1 s0
hdfs dfs -deleteSnapshot //fawzedestination s0
hdfs dfs -renameSnapshot //fawzedestination s1 s0
hdfs dfs -deleteSnapshot //fawzedestination s2
hdfs dfs -createSnapshot //fawzedestination s2
hadoop distcp -rdiff s2 s0 //fawzesource //fawzedestination
if [ $? -eq 0 ]
hdfs dfs -deleteSnapshot //fawzedestination s2
hdfs dfs -deleteSnapshot //fawzedestination s0
hdfs dfs -createSnapshot //fawzedestination s0