Member since
09-29-2015
56
Posts
8
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1279 | 06-08-2016 02:49 PM | |
662 | 04-14-2016 11:14 PM | |
1851 | 01-20-2016 03:22 AM | |
1101 | 12-21-2015 09:05 PM | |
1264 | 12-15-2015 10:55 PM |
06-08-2016
06:28 PM
@Piotr Pruski How are you setting up replication? Is it using Feed Replication or is it using Falcon recipe. Can you please do "falcon admin -version" and share the result. Can you also share the feed entity ?
... View more
06-08-2016
02:49 PM
Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.
... View more
05-19-2016
08:44 PM
The working and staging dir should have the following permissions. hadoop fs -chown -R falcon /user/falcon/sbpRndCluster/ hadoop fs -chmod -R 777 /user/falcon/sbpRndCluster/staging hadoop fs -chmod -R 755 /user/falcon/sbpRndCluster/working Once you set these permissions, submitting cluster should be straightforward. When you see errors like "Unable to validate the location with path: /apps/falcon/primaryCluster/staging for cluster:sbpRndCluster due to transient failures", can you please share the logs from falcon.application.log, this will give us more information about what is happening.
... View more
05-17-2016
04:14 PM
@Hanna Lee Please make sure the execute endpoint is using port 8050. You can find this in Yarn configs under yarn.resourcemanager.address. This port can be 8032 on some nodes, but I think sandbox uses 8050. Once you have the right port, please restart falcon and try submitting cluster again. The restart is required because there could be a stale FileSystem object in Falcon server.
... View more
04-14-2016
11:14 PM
To add to what @vramachandran said, Falcon also supports replication to Azure ( http://hortonworks.com/hadoop-tutorial/incremental-backup-data-hdp-azure-disaster-recovery-burst-capacity/) and AWS as well.
... View more
02-02-2016
09:49 PM
Ah, my bad. Too many messages here and I missed your solution. Thanks @Sowmya Ramesh
... View more
02-02-2016
09:40 PM
@Nayan Paul : I might have found the problem. Here is what you have for the process. <validity start="2015-12-01T23:33Z" end="2018-01-03T23:33Z"/> </cluster> ...
<parallel>1</parallel> <order>FIFO</order> <frequency>minutes(5)</frequency>
The input feed you have is 2016-01, almost a month after the validity start 2015-12. You have one process instance running every 5 minutes, so that is approximately 72x31 instances that should be processed before getting to 2016-01. So your process instances in 2015-12 are waiting for input feed 2015-12.... and will wait for almost a month since you said FIFO order should be used.
Create input feed 2015-12 and you will see data being processed end to end. Let me know if this solves the issue.
... View more
02-01-2016
10:51 PM
@Ashnee Sharma : Did @Kuldeep Kulkarni's response answer your question? If yes, please accept the answer
... View more
01-21-2016
07:22 PM
@bsaini , @Artem Ervits
... View more
01-21-2016
07:21 PM
Hi Team, The information in 2.3.0 docs had an error. Please use information at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/configuring_oozie_for_falcon.html , I will work with documentation team to get 2.3.0 doc fixed.
... View more
01-21-2016
07:20 PM
The entry is incorrect. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/configuring_oozie_for_falcon.html has the correct instructions. The line after dataIn should be as follows. instanceTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime,
... View more
01-20-2016
03:35 AM
We tested the 2.3 sandbox to make sure Oozie was configured correctly for Falcon. This seems to be a regression in latest Sandbox. Let me follow up with the team. Can you please share the Sandbox build you are using?
... View more
01-20-2016
03:22 AM
2 Kudos
This looks like a configuration issue where Oozie is probably not setup correctly for Falcon in the sandbox (Assuming you are using sandbox). Please refer to http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/configuring_oozie_for_falcon.html This document should explain all the configs Oozie should have for Falcon to work successfully. My guess is that the Coord EL Functions Properties are missing.
... View more
12-22-2015
09:46 PM
You are very welcome, Happy to be of help.
... View more
12-22-2015
09:46 PM
Unfortunately, there is no Falcon way to do this. Sorry.
... View more
12-21-2015
09:05 PM
1 Kudo
In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works. <availabilityFlag>_SUCCESS</availabilityFlag>
... View more
12-18-2015
04:39 PM
Yes. That is correct.
... View more
12-18-2015
03:45 PM
Hi, the error clearly says you did not give 777 permissions. Can you please try this command? hadoop fs -chmod -R 777 /apps/falcon/backupCluster/staging
... View more
12-16-2015
06:22 PM
The error says ... 61 more Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hdpexeusr, access=WRITE, inode="/apps/falcon/backupCluster/staging/falcon/workflows/feed":falcon:hdfs:drwxr-xr-x It is not sufficient to give perms to just the feed or process dir under /apps/falcon/backupCluster/staging/falcon/workflows , you must have permission 777 for ALL directories under /apps/falcon/backupCluster/staging.
... View more
12-15-2015
10:55 PM
Please refer to http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-... specifically the section about setting proxy user settings in oozie for falcon. 4. Add the following properties to custom oozie-site , your set up has it in custom core-site. Then restart oozie and you should be able to submit the cluster entity. <property>
<name>oozie.service.ProxyUserService.proxyuser.falcon.hosts</name>
<value>*</value>
</property>
<property>
<name>oozie.service.ProxyUserService.proxyuser.falcon.groups</name>
<value>*</value>
</property>
... View more
12-15-2015
09:20 PM
Hi Jeden : When you see an error like this, it means that Falcon server is unable to contact oozie server. Can you please try "curl from the falcon server and let us know if it works. You can get further information about the underlying exception if you look at falcon.application.log. Please share that exception as well, and we will be able to help you.
... View more
12-15-2015
07:41 PM
Hi Venkata, Can you please go to the falcon.application.log file, and paste the full exception you see when you attempt to submit the feed? This way, we can debug better.
... View more
12-11-2015
09:47 PM
Hi this is a known bug https://issues.apache.org/jira/browse/FALCON-1647. The workaround until we release next version of Falcon is to give 777 permissions to all directories under /apps/falcon/backupCluster/staging/falcon.
... View more
12-07-2015
11:19 PM
If the nameservice is "myHA", the interfaces should be "hdfs://myHA".
... View more
12-03-2015
10:37 PM
Unfortunately, I don't have any thoughts. We should follow up with Atlas. Qq : Does this happen with Falcon/Oozie logs as well?
... View more
12-03-2015
03:49 PM
My understanding is that cluster/YARN overload and this issue are not related.
... View more
12-02-2015
10:28 PM
1 Kudo
@Pavel Benes The /apps/falcon-MiddleGate/staging/ and /apps/falcon-MiddleGate/working dirs are created when the cluster entity is submitted by the user. These dirs are used to store Falcon specific information, staging dir should have permissions of 777 and working dirs should have permissions of 755. Falcon expects that in real usecases, a falcon cluster entity is created by the admin and feed/process entities are created by the users of the cluster. 1. why the permission for the 'feed' folder is now 'drwxr-xr-x' ? -- Falcon creates <staging_dir>/falcon/workflows/feed and <staging_dir>/falcon/workflows/process only when a feed/process entity are scheduled. The owner of these dirs is the user scheduling the entity. The permissions are based on the default umask of the FS. 2. I am inclined to agree with you. <staging_dir>/falcon/workflows/process and <staging_dir>/falcon/workflows/feed should be created when cluster entity is submitted, and the ownership should belong to falcon, with perms 777. I created a Jira https://issues.apache.org/jira/browse/FALCON-1647 and I will update/resolve it after discussing with Falcon community. The temporary workaround for this problem is to manually change the permissions of all dirs upto <staging_dir>/falcon/workflows/process and <staging_dir>/falcon/workflows/feed to 777 as @peeyush suggested.
... View more
11-30-2015
07:32 PM
Here is a sample feed xml. <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="rawEmailFeed" description="Raw customer email feed" xmlns="uri:falcon:feed:0.1">
<tags>externalSystem=USWestEmailServers</tags>
<groups>churnAnalysisDataPipeline</groups>
<frequency>hours(1)</frequency>
<timezone>UTC</timezone>
<late-arrival cut-off="hours(1)"/>
<clusters>
<cluster name="primaryCluster" type="source">
<validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
<retention limit="hours(10)" action="delete"/>
</cluster>
</clusters>
<locations>
<location type="data" path="/user/ambari-qa/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
<location type="stats" path="/"/>
<location type="meta" path="/"/>
</locations>
<ACL owner="ambari-qa" group="users" permission="0x755"/>
<schema location="/none" provider="/none"/>
</feed>
In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. You are correct when you say that some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted. If you are expecting all feed instances to be deleted after retention time limit, Falcon should change the way it creates retention coordinator job. I opened a Jira https://issues.apache.org/jira/browse/FALCON-1644 for this.
... View more
10-27-2015
10:48 PM
@Sowmya Ramesh Very good and detailed answer, thank you.
... View more
10-24-2015
12:26 AM
The issue raised by @Jeremy Dyer will be fixed by Falcon team in Dal-M20 release.
... View more