Reply
Expert Contributor
Posts: 158
Registered: ‎01-25-2017

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

@Yongjun Zhang@Yufei Gu

 

Hi,

 

Do you think it's a normal behaviour or a BUG?

Cloudera Employee
Posts: 13
Registered: ‎08-20-2015

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

HI Fawze,

If distcp failed, the reason need to be understood. The side effect of
failed distcp is normal.

Thanks.

--Yongjun
Expert Contributor
Posts: 158
Registered: ‎01-25-2017

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

Hi @Yongjun Zhang

 

 

Simple reason for the failed distcp could be interrupting the process or any network issue.

 

what do you mean that the side effects is normal? distcp should fail all the time when distcp using snap**bleep** failed? 

Explorer
Posts: 7
Registered: ‎05-04-2015

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

Yongjun

 

Our production DCs replication process (based on distcp) has stopped working after upgrading to CDH 5.10. 

We urged to do a rollback to CDH 5.9.  

It's a really critical problem for us  

Please help us to resolve this.

Thank you,

Vladi

 

Cloudera Employee
Posts: 13
Registered: ‎08-20-2015

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

Hi Fawze, 
 
What I meant by "side effect" is that with partially finished distcp job, some files are copied, some are not.
 

Hi Vladi,

 
Sorry to hear that. I'd suggest that we understand the failure reason first to tell if it's because 5.10 upgrade or something else.  Would you please describe the failure symptom and report a case to our support so that we can keep track? 

Have we tried HDFS-9820 feature as we talked earlier?
 
Thanks.
Explorer
Posts: 22
Registered: ‎04-12-2016

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

Hi,

We don't have a license to open a case to support, and I posted the error with debug level in the previous comments, and the error is so vague with less information that I can get any clue why it was failed.

To use the new feature it need time be build a solution and test it, and I asked a help also in the previous comments and I had few questions that I asked.

Had you a chance to check my previous comments?
Expert Contributor
Posts: 158
Registered: ‎01-25-2017

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

@Yongjun Zhang 

 

Hi,

 

Any chance to help here?

Cloudera Employee
Posts: 13
Registered: ‎08-20-2015

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

Sorry guys for delayed reply.

 

We can use the HDFS-9820 feature as 

 
After applying HDFS-9820 (-rdiff, usage see above links), we have made the target cluster's current state to the base snapshot s0 at the source. We can create snapshot s0 at the target now, then apply -diff to copy the delta over from source to target.
 
Thanks.
 
Expert Contributor
Posts: 158
Registered: ‎01-25-2017

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

Seems the documentation is misleading .... and you didn't get the time to
read my comments and answer my questions.

Thanks for your help and i will give it a last try and if it will not work,
we will look for another solution other than the distcp.
Cloudera Employee
Posts: 13
Registered: ‎08-20-2015

Re: Killing the Distcp which running over snapshot listing all snapshottable path in the next run

Sorry for the confusion. I hoped the two links I provided have the answers. Would you please read the two links I shared earlier, and point out the confusing statements so we can correct?

 

1. Should i use rdiff only if distcp fail or just replaced the current diff with it, if it only when distco failed so my script will have

A:

Hopefully this is clear from the documentation. The purpose of "-diff s0 s1" is to go from s0 to s1, "-ridff s1 s0" is to go from s1 to s0. s0 is older than s1. So it's not a replacement. 

 

2. Do you know if i should manage the create/delete/ and update snapshots
during the distcp or it will be as a part of the restore process?

A: You need to manage the creation/deletion of snapshots yourself, or you can use cloudera BDR which manages it for you.

 

Thanks.

 

Announcements