Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Falcon Doesn't Mirror Data?

avatar
Expert Contributor

Hi,

I set up two HDP clusters and tried to use Falcon to mirror data from one cluster to another.

I could created two clusters and configured the mirror setting on Falcon UI without error. I could also see the mirroring MR job completed successfully. However, files do not get mirrored from source to target.

I am attaching my Falcon Mirror setting. Am I missing anything?

4662-falcon-mirror.png

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Thanks @yzheng

The mirror instance is in healthy status.

After upgraded the target cluster to 2.3.4 and changed to job to run on source cluster, it works now.

View solution in original post

8 REPLIES 8

avatar
Super Collaborator

@yjiang What is the HDP version you are using, for which you have seen that Falcon does not mirror data.

avatar
Expert Contributor

@peeyush That is the target cluster running HDP 2.3.4.7-4. The source cluster is HDP 2.4.2.0

avatar
Super Guru
@Rahul Pathakcan you help here ?

avatar
Contributor

@yjiang Could you check the mirror instance status? You can run CLI command

bin/falcon instance -list -type process -name [your-mirror-name]

If it is failed or killed, could you check Oozie site on error details?

avatar
Expert Contributor

Thanks @yzheng

The mirror instance is in healthy status.

After upgraded the target cluster to 2.3.4 and changed to job to run on source cluster, it works now.

avatar
Contributor

@yjiang Great!

avatar
Master Guru

There is a bug in hdfs replication workflow FALCON-189 causing this. As a quick fix, edit drSourceDir to include source FS, so that it become: ${drSourceClusterFS}${drSourceDir} as discussed in the Jira. The file to fix is

/usr/hdp/current/falcon-server/data-mirroring/workflows/hdfs-replication-workflow.xml

on your Falcon server. Then upload the file to /apps/data-mirroring/workflows in HDFS. Restart Falcon and retry to run mirroring from the target cluster.

avatar
Contributor

Upload the hdfs-replication-workflow.xml into hdfs will resolve this problem.

i.e. create a hdfs directory /apps/data-mirroring/workflows and upload it