I setup Falcon Mirroring as outlined in
But mirroring is not happening.
Only difference between the document and my case is that I am trying to mirror between two different clusters (while I think the document is trying to replicate within the same cluster).
I have been looking at the issue from various angles from yesterday - but no luck yet.
The oozie logs still show as disabled - even though I have done the fix as specified by Hortonworks.
I also tried the fix as in :
But still no replication is happening.
Appreciate any insights on fixing this.
These are the permissions on the source and target dirs. :
[ambari-qa@hdtesting1 ~]$ hdfs dfs -ls -R /user/ambari-qa/falcon drwxrwxrwx - ambari-qa hdfs 0 2017-03-07 09:20 /user/ambari-qa/falcon/mirrorSrc -rw-r--r-- 3 ambari-qa hdfs 2 2017-03-07 09:20 /user/ambari-qa/falcon/mirrorSrc/testfile -rwxrwxrwx 3 ambari-qa hdfs 2048 2017-03-06 10:20 /user/ambari-qa/falcon/mirrorSrc/weekly-time-entry.txt drwxrwxrwx - ambari-qa hdfs 0 2017-03-06 09:07 /user/ambari-qa/falcon/mirrorTgt
[ambari-qa@hdtesting5 ~]$ hdfs dfs -ls /user/ambari-qa/falcon/ Found 2 items drwxrwxrwx - ambari-qa hdfs 0 2017-03-06 09:17 /user/ambari-qa/falcon/mirrorSrc drwxrwxrwx - ambari-qa hdfs 0 2017-03-06 09:09 /user/ambari-qa/falcon/mirrorTgt
I tried to replicate to the same cluster - even that is not working.
Distcp to the same cluster or across two clusters works fine.
I checked the falcon and oozie logs - do not see any specific Error.
Source and Target dirs. have the right path :
But whatever files I moved in for mirroring - I don't see any mention of them in the logs - which leads me to believe that falcon is not seeing them.
I see this in the falcon logs :
No run dirs are available in logs dir:
Anything else I should look for in the log ...
Use Falcon CLI to make sure Falcon cluster and other entitites have been correctly created, for example, as user falcon: "falcon entity -type cluster -list" and "falcon entity -type cluster -name <cluster1-ent-name> -definition". If all that is fine schedule a mirroring, and monitor Oozie workflows, there should be 2 of them started. Inspect Oozie logs either from Ooze Web UI, or, if nothing appears there, get detailed logs using "oozie job -log <workflow-id>". Details on Falcon CLI here. Also make sure you create mirroring entities using the same user for which distcp successfully ran.
I checked and found that the falcon definitions were created with falcon user. So I recreated the definitions as ambari-qa - but still didn't work.
Here are the definitions - if you can spot something wrong :
Name: falconcluster1 Colo: primary colo Description: this is primary cluster ACL Owner: admin Group: users Permission: 0755 Interfaces readonly: hftp://hdtesting4.co.local:50070 - 2.2.0 write: hdfs://hdtesting4.co.local:8020 - 2.2.0 execute: hdtesting4.co.local:8050 - 2.2.0 workflow: http://hdtesting2.co.local:11000/oozie/ - 4.0.0 messaging: tcp://hdtesting2.co.local:61616?daemon=true - 5.1.6 Locations staging: /apps/falcon/primaryCluster/staging temp: /tmp working: /apps/falcon/primaryCluster/working
Name: backupCluster Colo: backupColo Description: this is the backup cluster ACL Owner: admin Group: users Permission: 0755 Interfaces readonly: hftp://hdtesting5.co.local:50070 - 2.2.0 write: hdfs://hdtesting5.co.local:8020 - 2.2.0 execute: hdtesting6.co.local:8050 - 2.2.0 workflow: http://hdtesting6.co.local:11000/oozie/ - 4.0.0 messaging: tcp://hdtesting6.co.local:61616?daemon=true - 5.1.6 Locations staging: /apps/falcon/backupCluster/staging temp: /tmp working: /apps/falcon/backupCluster/working
If I try to see the Job Error Log for the mirror job I see :
Error Log is disabled!!
I did this :
But still I can't see the log.
>monitor Oozie workflows, there should be 2 of them started
In Falcon UI, if I click on the job I can see a bunch of instances (?) of the job (with date and ending with OZ) with only one RUNNING and the rest WAITING.
I see a /var/log/falcon/falcon.application.log and a bunch of logs in /var/log/oozie - is there no meaningful way to check if it polled the source directory, what files it found, what it did with them?