Member since
09-29-2015
56
Posts
8
Kudos Received
6
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
524 | 06-08-2016 02:49 PM | |
264 | 04-14-2016 11:14 PM | |
682 | 01-20-2016 03:22 AM | |
380 | 12-21-2015 09:05 PM | |
468 | 12-15-2015 10:55 PM |
06-08-2016
06:28 PM
@Piotr Pruski How are you setting up replication? Is it using Feed Replication or is it using Falcon recipe. Can you please do "falcon admin -version" and share the result. Can you also share the feed entity ?
... View more
06-08-2016
03:29 PM
@Rahul Pathak gave the correct answer. Apart from setting up NN ha config in core-site.xml, user also has to set property oozie.service.HadoopAccessorService.hadoop.configurations on the Oozie server.
... View more
06-08-2016
03:26 PM
Apart from defining location for data, can you please try setting "stats" and "meta" to "/tmp/${YEAR}-${MONTH}-${DAY}" ? Please let us know if this works. Also, please let me know the Java version you are using, I suspect the change in behavior is due to possible change in how pattern matcher behaves between Java 7 and 8.
... View more
06-08-2016
03:22 PM
This is similar to https://community.hortonworks.com/questions/17000/feed-unable-to-resolve-pattern-for-feedpath.html
... View more
06-08-2016
02:57 PM
@Misael Castro : If the process is not ending, here are a few things you should check. 1. What state is the process instance in Oozie. If it is in "WAITING", this is probably because you do not have sufficient containers/memory in Yarn to run the job. Please configure yarn accordingly. 2. If the process instance is in RUNNING state, please look into Yarn logs and see why the Pig script is not completing. Let me know if it is 1 or 2, and I can try to help you with configuration.
... View more
06-08-2016
02:49 PM
Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.
... View more
05-26-2016
03:23 PM
You can find logs by clicking on console url and clicking on the "1" under map tasks. I request you to share the hadoop version you are using.
... View more
05-26-2016
03:21 PM
@Junaid Rao
and @mayki wogno : What does your logs in Yarn say? Do you see an error similar to this? Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306)
at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
... 15 more
Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /
at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:518)
at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331)
at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300)
... 23 more
... View more
05-19-2016
08:51 PM
If you look at the XML on the right hand side (in the red oval)4309-falcon-edit-xml-datapath.png, you will notice that location tag for primary cluster does not have an entry for "path". Please fix this and you should be able to submit the cluster.
... View more
05-19-2016
08:44 PM
The working and staging dir should have the following permissions. hadoop fs -chown -R falcon /user/falcon/sbpRndCluster/ hadoop fs -chmod -R 777 /user/falcon/sbpRndCluster/staging hadoop fs -chmod -R 755 /user/falcon/sbpRndCluster/working Once you set these permissions, submitting cluster should be straightforward. When you see errors like "Unable to validate the location with path: /apps/falcon/primaryCluster/staging for cluster:sbpRndCluster due to transient failures", can you please share the logs from falcon.application.log, this will give us more information about what is happening.
... View more
05-19-2016
08:40 PM
@mayki wogno : You did all the steps correct. The only possible explanation for you to continue seeing this error is that hdfs, yarn and oozie were not restarted after the changes were made to configs. If you did restart hdfs, yarn and oozie - I am not sure why you are continuing to see the problem.
Based on the logs, I am assuming you have enabled Kerberos on your cluster. I see that the job is scheduled from clusterA, but the workflow is on clusterB. Oozie user oozie/clusterA@EXAMPLE.COM is unable to connect to clusterB and read the workflow.xml. This seems like kerberos configuration issue. Make sure, you have the config "oozie.authentication.kerberos.name.rules" set up correctly to handle these situations, and that kerberos server on clusterB can authenticate oozie/clusterA@EXAMPLE.COM
... View more
05-17-2016
04:35 PM
@mayki wogno Please share the relevant Oozie log for this workflow. This seems like auth issue on Oozie and not Falcon, and I will help you debug.
... View more
05-17-2016
04:24 PM
@Manoj Dhake Can you please confirm that oozie is configured correctly for Falcon to work? You can follow doc at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/configuring_oozie_for_falcon.html
... View more
05-17-2016
04:22 PM
@mayki wogno Please go to Ambari, click on Falcon service, now go to configs and go to "Advanced falcon-env". This is where the log dir is configured for falcon. The log name should be falcon.application.log
... View more
05-17-2016
04:14 PM
@Hanna Lee Please make sure the execute endpoint is using port 8050. You can find this in Yarn configs under yarn.resourcemanager.address. This port can be 8032 on some nodes, but I think sandbox uses 8050. Once you have the right port, please restart falcon and try submitting cluster again. The restart is required because there could be a stale FileSystem object in Falcon server.
... View more
05-17-2016
03:54 PM
1 Kudo
The root cause of this issue is that the user you are using to run the workflow does not have permission to read the workflow.xml. If you submitted the feed entity as user "ABC", try to schedule it as user "ABC" and not some other user who does not have permissions to read the workflow.
... View more
04-15-2016
01:01 AM
Shouldnt the uri look like jdbc:hive2://<host>:<port>/<db>;transportMode=http;httpPath=<http_endpoint> ?
... View more
04-14-2016
11:14 PM
To add to what @vramachandran said, Falcon also supports replication to Azure ( http://hortonworks.com/hadoop-tutorial/incremental-backup-data-hdp-azure-disaster-recovery-burst-capacity/) and AWS as well.
... View more
04-13-2016
11:17 PM
@Saurabh Kumar I did not refer to any feed/process entities in my comment. So I do not understand your comment. Can you please clarify?
... View more
04-12-2016
11:40 PM
2 Kudos
https://falcon.apache.org/HiveDR.html explains how you can setup Hive replication from DB to DB, Table(s) to Table(s). Here are the limitations. 1. The DB and Table(s) names should be same on source and target. 2. Make sure hive has property hive.metastore.event.listeners set to org.apache.hive.hcatalog.listener.DbNotificationListener 3. User should bootstrap the DB and Table(s) on target from the source, after enabling event listeners. As of today, Falcon will only replicate events captured by hive.metastore.event.listeners. This means that for Hive 1.2.*, the following will NOT be replicated. Everything else should be replicated by Falcon because Hive saves the events. - will not replicate virtual objects like views, and will not handle other metadata objects such as roles, etc, it will be up to the warehouse administrator to manage those aspects currently. - will not replicate direct HDFS writes without Metadata registering (in Alter, Update scenarios)
... View more
03-07-2016
11:12 PM
@wood super : I tried running falcon-0.9 in embedded mode. The UI worked fine without any problems. Can you please share the logs in falcon.application.log when you see the "no found" error? Can you also let us know if this is secure cluster or un-secure cluster?
... View more
02-02-2016
09:49 PM
Ah, my bad. Too many messages here and I missed your solution. Thanks @Sowmya Ramesh
... View more
02-02-2016
09:40 PM
@Nayan Paul : I might have found the problem. Here is what you have for the process. <validity start="2015-12-01T23:33Z" end="2018-01-03T23:33Z"/> </cluster> ...
<parallel>1</parallel> <order>FIFO</order> <frequency>minutes(5)</frequency>
The input feed you have is 2016-01, almost a month after the validity start 2015-12. You have one process instance running every 5 minutes, so that is approximately 72x31 instances that should be processed before getting to 2016-01. So your process instances in 2015-12 are waiting for input feed 2015-12.... and will wait for almost a month since you said FIFO order should be used.
Create input feed 2015-12 and you will see data being processed end to end. Let me know if this solves the issue.
... View more
02-02-2016
09:20 PM
Hi @Hanna Lee : Select the instance, and click on Log. You should be able to see basic logs on why this instance failed. You can get the job-id from the log, and look directly into Oozie to get more information. Please let me know what the log says.
... View more
02-01-2016
10:53 PM
@Rahul Pathak is correct. @Hanna Lee : Please follow instructions at Please use information at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/configuring_oozie_for_falcon.html and ensure Oozie is set up correctly for Falcon. Please let us know if this solved the issue.
... View more
02-01-2016
10:51 PM
@Ashnee Sharma : Did @Kuldeep Kulkarni's response answer your question? If yes, please accept the answer
... View more
01-21-2016
07:22 PM
@bsaini , @Artem Ervits
... View more
01-21-2016
07:21 PM
Hi Team, The information in 2.3.0 docs had an error. Please use information at https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/configuring_oozie_for_falcon.html , I will work with documentation team to get 2.3.0 doc fixed.
... View more
01-21-2016
07:20 PM
The entry is incorrect. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_installing_manually_book/content/configuring_oozie_for_falcon.html has the correct instructions. The line after dataIn should be as follows. instanceTime=org.apache.oozie.coord.CoordELFunctions#ph3_coord_nominalTime,
... View more
01-20-2016
03:35 AM
We tested the 2.3 sandbox to make sure Oozie was configured correctly for Falcon. This seems to be a regression in latest Sandbox. Let me follow up with the team. Can you please share the Sandbox build you are using?
... View more