- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Feed : Unable to resolve pattern for feedPath
- Labels:
-
Apache Falcon
Created ‎02-15-2016 10:15 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I get this error when I run a Feed. My feed is running but when i check into the cluster I find this in the log :
org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306) at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56) ... 15 more Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /user/ambari-qa/falcon at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:435) at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331) at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300) ... 23 more
I don't really know what does this mean and How do i fix this ?
Am I right when I expect to see the file of the dataPath I provide (/usr/ambari-qa/falcon) in my cluster ?
Created ‎06-08-2016 02:49 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.
Created ‎02-15-2016 10:24 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you add your feed xml? There should be a location something like this:
<locations> <location type="data" path="/mydata/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> </locations>
Normally feeds are parametrized to correspond to the frequency of the feed.
So my question is, does the folder exist in HDFS? Do you have any retention policy? Eviction sounds like that and would not work in a folder without a time component. Easiest way would be to upload your feed xml.
Created ‎02-15-2016 10:33 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Here is my feed xml
<feed xmlns='uri:falcon:feed:0.1' name='testFeed2'> <frequency>minutes(5)</frequency> <timezone>GMT+01:00</timezone> <clusters> <cluster name='primaryCluster' type='source'> <validity start='2016-02-15T08:41Z' end='2016-02-16T09:41Z'/> <retention limit='minutes(5)' action='delete'/> <locations> <location type='data'> </location> <location type='stats'> </location> <location type='meta'> </location> </locations> </cluster> </clusters> <locations> <location type='data' path='/user/ambari-qa/falcon/'> </location> <location type='stats' path='/'> </location> <location type='meta' path='/'> </location> </locations> <ACL owner='ambari-qa' group='users' permission='0x755'/> <schema location='/none' provider='/none'/> <properties> <property name='jobPriority' value='NORMAL'> </property> </properties> </feed>
An I am sure the data path exist :
-rw-r--r-- 3 hdfs hdfs 0 2016-02-15 09:46 /user/ambari-qa/falcon/test -rwxrwxrwx 3 ambari-qa hdfs 0 2016-02-15 09:33 /user/ambari-qa/falcon/testFeed
I don't understand why I need a time component ?
Created ‎02-15-2016 10:40 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You have a retention period for your feed. Which means Falcon comes up and tries to delete all folders that belong to feed instances that are older than 5 minutes.
So he expects something like:
path='/inputfolder/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}'
So folders in HDFS that look like
/inputfolder/2016-01-01-09-15
etc. pp. And then he can delete all folders that are older than that.
It is all very nicely explained in the oozie documentation ( Falcon is build on oozie )
http://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a5._Dataset
Created ‎02-19-2016 09:26 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Benjamin Leonhardi
I still have the same error even if when I change the data path :
org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
And the feed is :
<feed xmlns='uri:falcon:feed:0.1' name='Depot'> <tags>projet=103,data=raw</tags> <frequency>hours(1)</frequency> <timezone>UTC</timezone> <clusters> <cluster name='primaryCluster' type='source'> <validity start='2016-02-18T10:11Z' end='2016-02-19T10:11Z'/> <retention limit='hours(4)' action='delete'/> <locations> <location type='data'> </location> <location type='stats'> </location> <location type='meta'> </location> </locations> </cluster> </clusters> <locations> <location type='data' path='/user/hdfs/falcon/DepotProject103/${YEAR}-${MONTH}-${DAY}-${HOUR}'> </location> <location type='stats' path='/'> </location> <location type='meta' path='/'> </location> </locations> <ACL owner='ambari-qa' group='users' permission='0x755'/> <schema location='/none' provider='/none'/> <properties> <property name='jobPriority' value='NORMAL'> </property> </properties> </feed>
I think I did it right this time ? Is there something I am missing again ?
Created ‎06-08-2016 02:42 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Whats the solution to this problem ??
Created ‎06-08-2016 02:49 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.
Created ‎06-09-2016 07:34 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Balu : I changed the "stats" & "meta" location in Feed as you mentioned....that error got resolved & eviction got succeeded..thanks for the answer
