question Re: DR with Falcon: handling changing data; distcp validation; snapshotting in Archives of Support Questions (Read Only)

DR with Falcon: handling changing data; distcp validation; snapshotting

ppruski — Wed, 04 May 2016 22:35:03 GMT

Looking for best practises around DR replication option with Falcon (or Ozzie+distcp..).

Using either feed based replication or mirror recipe in Falcon (that both leverage distcp to my understanding), how does it handle the situation where clients are still writing, moving, or deleting in the source cluster?
The distcp documentation states if another client is still writing to a source file, the copy will likely fail..
Does Falcon provide any data validation mechanism that the transfer with distcp was successful?
What additional benefit would snapshotting have here? (and does Falcon do this?)

divakarreddy_a — Wed, 04 May 2016 22:42:05 GMT

I think we have answers here for your 1 or 2 questions. link

rbiswas1 — Mon, 09 May 2016 12:39:21 GMT

@piotr pruski

Nice question. Would help a lot of people in the community.

Thank you.

sramesh — Tue, 10 May 2016 01:42:20 GMT

As you mentioned Falcon piggy backs on DistCP under the hood to achieve replication. If another client is still writing to a source file, the copy will likely fail
If the DistCP job fails then Falcon replication job fails too and status API/command can be used to get the finished status of the replication job. Same in case of success too. Also with FALCON-1313 support was added for email based notification for job status for Feeds and mirror recipes.
Replication using snapshots is not yet supported in Falcon. This feature is added with FALCON-1861. Additional benefit is performance. It leverages HDFS snapshots which are very cost effective to create ( cost is O(1) excluding inode lookup time). Once created, it is very efficient to find modifications relative to a snapshot and copy over these modifications for disaster recovery (DR). This makes it's cost effective.

cnormile — Thu, 12 May 2016 01:17:34 GMT

To add to Sowmya's response:

If a Falcon Mirror process fails, Falcon will attempt the copy again, so a file momentarily open will be captured on the retry.

ambud_sharma1 — Sat, 19 Nov 2016 02:53:37 GMT

What about support for -overwrite and -update flags in HDFS Mirror (Falcon) ?