Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Falcon Doesn't Mirror Data?

avatar
Expert Contributor

Hi,

I set up two HDP clusters and tried to use Falcon to mirror data from one cluster to another.

I could created two clusters and configured the mirror setting on Falcon UI without error. I could also see the mirroring MR job completed successfully. However, files do not get mirrored from source to target.

I am attaching my Falcon Mirror setting. Am I missing anything?

4662-falcon-mirror.png

1 ACCEPTED SOLUTION

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
8 REPLIES 8

avatar
Super Collaborator

@yjiang What is the HDP version you are using, for which you have seen that Falcon does not mirror data.

avatar
Expert Contributor

@peeyush That is the target cluster running HDP 2.3.4.7-4. The source cluster is HDP 2.4.2.0

avatar
Super Guru
@Rahul Pathakcan you help here ?

avatar
Contributor

@yjiang Could you check the mirror instance status? You can run CLI command

bin/falcon instance -list -type process -name [your-mirror-name]

If it is failed or killed, could you check Oozie site on error details?

avatar
Expert Contributor
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Contributor

@yjiang Great!

avatar
Master Guru

There is a bug in hdfs replication workflow FALCON-189 causing this. As a quick fix, edit drSourceDir to include source FS, so that it become: ${drSourceClusterFS}${drSourceDir} as discussed in the Jira. The file to fix is

/usr/hdp/current/falcon-server/data-mirroring/workflows/hdfs-replication-workflow.xml

on your Falcon server. Then upload the file to /apps/data-mirroring/workflows in HDFS. Restart Falcon and retry to run mirroring from the target cluster.

avatar
Contributor

Upload the hdfs-replication-workflow.xml into hdfs will resolve this problem.

i.e. create a hdfs directory /apps/data-mirroring/workflows and upload it