Simple reason for the failed distcp could be interrupting the process or any network issue.
what do you mean that the side effects is normal? distcp should fail all the time when distcp using snap**bleep** failed?
Our production DCs replication process (based on distcp) has stopped working after upgrading to CDH 5.10.
We urged to do a rollback to CDH 5.9.
It's a really critical problem for us
Please help us to resolve this.
Sorry guys for delayed reply.
We can use the HDFS-9820 feature as
Sorry for the confusion. I hoped the two links I provided have the answers. Would you please read the two links I shared earlier, and point out the confusing statements so we can correct?
1. Should i use rdiff only if distcp fail or just replaced the current diff with it, if it only when distco failed so my script will have
Hopefully this is clear from the documentation. The purpose of "-diff s0 s1" is to go from s0 to s1, "-ridff s1 s0" is to go from s1 to s0. s0 is older than s1. So it's not a replacement.
2. Do you know if i should manage the create/delete/ and update snapshots
during the distcp or it will be as a part of the restore process?
A: You need to manage the creation/deletion of snapshots yourself, or you can use cloudera BDR which manages it for you.