Support Questions
Find answers, ask questions, and share your expertise

FALCON HDFS Mirroring not working

FALCON HDFS Mirroring not working

Explorer

I setup Falcon Mirroring as outlined in

https://hortonworks.com/hadoop-tutorial/create-falcon-cluster/

and

https://hortonworks.com/hadoop-tutorial/mirroring-datasets-between-hadoop-clusters-with-apache-falco...

But mirroring is not happening.

Only difference between the document and my case is that I am trying to mirror between two different clusters (while I think the document is trying to replicate within the same cluster).

I have been looking at the issue from various angles from yesterday - but no luck yet.

The oozie logs still show as disabled - even though I have done the fix as specified by Hortonworks.

I also tried the fix as in :

https://issues.apache.org/jira/browse/FALCON-1894

But still no replication is happening.

Appreciate any insights on fixing this.

31 REPLIES 31

Re: FALCON HDFS Mirroring not working

Explorer

These are the permissions on the source and target dirs. :

[ambari-qa@hdtesting1 ~]$ hdfs dfs -ls -R /user/ambari-qa/falcon drwxrwxrwx - ambari-qa hdfs 0 2017-03-07 09:20 /user/ambari-qa/falcon/mirrorSrc -rw-r--r-- 3 ambari-qa hdfs 2 2017-03-07 09:20 /user/ambari-qa/falcon/mirrorSrc/testfile -rwxrwxrwx 3 ambari-qa hdfs 2048 2017-03-06 10:20 /user/ambari-qa/falcon/mirrorSrc/weekly-time-entry.txt drwxrwxrwx - ambari-qa hdfs 0 2017-03-06 09:07 /user/ambari-qa/falcon/mirrorTgt

[ambari-qa@hdtesting5 ~]$ hdfs dfs -ls /user/ambari-qa/falcon/ Found 2 items drwxrwxrwx - ambari-qa hdfs 0 2017-03-06 09:17 /user/ambari-qa/falcon/mirrorSrc drwxrwxrwx - ambari-qa hdfs 0 2017-03-06 09:09 /user/ambari-qa/falcon/mirrorTgt

Re: FALCON HDFS Mirroring not working

Explorer

@Frank Lu appreciate any insights on how to debug the replication flow to figure out where it is going wrong. Thanks.

Re: FALCON HDFS Mirroring not working

Rising Star

Re: FALCON HDFS Mirroring not working

Explorer

I tried to replicate to the same cluster - even that is not working.

Distcp to the same cluster or across two clusters works fine.

I checked the falcon and oozie logs - do not see any specific Error.

Source and Target dirs. have the right path :

<name>drSourceDir</name> <value>/user/ambari-qa/falcon/mirrorSrc</value>

<name>drTargetDir</name> <value>/user/ambari-qa/falcon/mirrorTgt</value>

But whatever files I moved in for mirroring - I don't see any mention of them in the logs - which leads me to believe that falcon is not seeing them.

I see this in the falcon logs :

No run dirs are available in logs dir:

Anything else I should look for in the log ...

Re: FALCON HDFS Mirroring not working

Use Falcon CLI to make sure Falcon cluster and other entitites have been correctly created, for example, as user falcon: "falcon entity -type cluster -list" and "falcon entity -type cluster -name <cluster1-ent-name> -definition". If all that is fine schedule a mirroring, and monitor Oozie workflows, there should be 2 of them started. Inspect Oozie logs either from Ooze Web UI, or, if nothing appears there, get detailed logs using "oozie job -log <workflow-id>". Details on Falcon CLI here. Also make sure you create mirroring entities using the same user for which distcp successfully ran.

Re: FALCON HDFS Mirroring not working

Explorer

I checked and found that the falcon definitions were created with falcon user. So I recreated the definitions as ambari-qa - but still didn't work.

Here are the definitions - if you can spot something wrong :

Name: falconcluster1 Colo: primary colo Description: this is primary cluster ACL Owner: admin Group: users Permission: 0755 Interfaces readonly: hftp://hdtesting4.co.local:50070 - 2.2.0 write: hdfs://hdtesting4.co.local:8020 - 2.2.0 execute: hdtesting4.co.local:8050 - 2.2.0 workflow: http://hdtesting2.co.local:11000/oozie/ - 4.0.0 messaging: tcp://hdtesting2.co.local:61616?daemon=true - 5.1.6 Locations staging: /apps/falcon/primaryCluster/staging temp: /tmp working: /apps/falcon/primaryCluster/working

Name: backupCluster Colo: backupColo Description: this is the backup cluster ACL Owner: admin Group: users Permission: 0755 Interfaces readonly: hftp://hdtesting5.co.local:50070 - 2.2.0 write: hdfs://hdtesting5.co.local:8020 - 2.2.0 execute: hdtesting6.co.local:8050 - 2.2.0 workflow: http://hdtesting6.co.local:11000/oozie/ - 4.0.0 messaging: tcp://hdtesting6.co.local:61616?daemon=true - 5.1.6 Locations staging: /apps/falcon/backupCluster/staging temp: /tmp working: /apps/falcon/backupCluster/working

Re: FALCON HDFS Mirroring not working

Explorer

If I try to see the Job Error Log for the mirror job I see :

Error Log is disabled!!

I did this :

https://community.hortonworks.com/content/supportkb/49164/how-to-enable-job-error-log-in-oozie-so-er...

But still I can't see the log.

Re: FALCON HDFS Mirroring not working

Explorer

>monitor Oozie workflows, there should be 2 of them started

In Falcon UI, if I click on the job I can see a bunch of instances (?) of the job (with date and ending with OZ) with only one RUNNING and the rest WAITING.

Re: FALCON HDFS Mirroring not working

Explorer

I see a /var/log/falcon/falcon.application.log and a bunch of logs in /var/log/oozie - is there no meaningful way to check if it polled the source directory, what files it found, what it did with them?