question Re: Falcon Doesn't Mirror Data? in Archives of Support Questions (Read Only)

Falcon Doesn't Mirror Data?

yjiang — Mon, 19 Aug 2019 11:33:28 GMT

Hi,

I set up two HDP clusters and tried to use Falcon to mirror data from one cluster to another.

I could created two clusters and configured the mirror setting on Falcon UI without error. I could also see the mirroring MR job completed successfully. However, files do not get mirrored from source to target.

I am attaching my Falcon Mirror setting. Am I missing anything?

Re: Falcon Doesn't Mirror Data?

pbishnoi — Tue, 31 May 2016 01:16:24 GMT

@yjiang What is the HDP version you are using, for which you have seen that Falcon does not mirror data.

Re: Falcon Doesn't Mirror Data?

yjiang — Tue, 31 May 2016 06:21:44 GMT

@peeyush That is the target cluster running HDP 2.3.4.7-4. The source cluster is HDP 2.4.2.0

Re: Falcon Doesn't Mirror Data?

sshimpi — Tue, 31 May 2016 12:46:44 GMT

@Rahul Pathakcan you help here ?

Re: Falcon Doesn't Mirror Data?

yzheng — Wed, 01 Jun 2016 08:11:40 GMT

@yjiang Could you check the mirror instance status? You can run CLI command

bin/falcon instance -list -type process -name [your-mirror-name]

If it is failed or killed, could you check Oozie site on error details?

Re: Falcon Doesn't Mirror Data?

yjiang — Wed, 01 Jun 2016 15:24:51 GMT

Thanks @yzheng

The mirror instance is in healthy status.

After upgraded the target cluster to 2.3.4 and changed to job to run on source cluster, it works now.

Re: Falcon Doesn't Mirror Data?

yzheng — Wed, 01 Jun 2016 22:09:32 GMT

@yjiang Great!

Re: Falcon Doesn't Mirror Data?

pminovic — Thu, 02 Jun 2016 06:54:05 GMT

There is a bug in hdfs replication workflow FALCON-189 causing this. As a quick fix, edit drSourceDir to include source FS, so that it become: ${drSourceClusterFS}${drSourceDir} as discussed in the Jira. The file to fix is

/usr/hdp/current/falcon-server/data-mirroring/workflows/hdfs-replication-workflow.xml

on your Falcon server. Then upload the file to /apps/data-mirroring/workflows in HDFS. Restart Falcon and retry to run mirroring from the target cluster.

Re: Falcon Doesn't Mirror Data?

rbaskaran — Mon, 05 Dec 2016 09:47:16 GMT

Upload the hdfs-replication-workflow.xml into hdfs will resolve this problem.

i.e. create a hdfs directory /apps/data-mirroring/workflows and upload it