question Re: Feed : Unable to resolve pattern for feedPath in Support Questions

Feed : Unable to resolve pattern for feedPath

arthur_grevin — Mon, 15 Feb 2016 18:15:40 GMT

Hi,

I get this error when I run a Feed. My feed is running but when i check into the cluster I find this in the log :

org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
	at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306)
	at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
	... 15 more
Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /user/ambari-qa/falcon
	at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:435)
	at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331)
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300)
	... 23 more

I don't really know what does this mean and How do i fix this ?

Am I right when I expect to see the file of the dataPath I provide (/usr/ambari-qa/falcon) in my cluster ?

Re: Feed : Unable to resolve pattern for feedPath

bleonhardi — Mon, 15 Feb 2016 18:24:02 GMT

Can you add your feed xml? There should be a location something like this:

Normally feeds are parametrized to correspond to the frequency of the feed.

So my question is, does the folder exist in HDFS? Do you have any retention policy? Eviction sounds like that and would not work in a folder without a time component. Easiest way would be to upload your feed xml.

Re: Feed : Unable to resolve pattern for feedPath

arthur_grevin — Mon, 15 Feb 2016 18:33:51 GMT

@Benjamin Leonhardi

Here is my feed xml

<feed xmlns='uri:falcon:feed:0.1' name='testFeed2'>
  <frequency>minutes(5)</frequency>
  <timezone>GMT+01:00</timezone> 
  <clusters>
    <cluster name='primaryCluster' type='source'> 
      <validity start='2016-02-15T08:41Z' end='2016-02-16T09:41Z'/>
      <retention limit='minutes(5)' action='delete'/>
      <locations>
        <location type='data'> 
        </location>
        <location type='stats'>
        </location>
        <location type='meta'> 
        </location>
      </locations>
    </cluster> 
  </clusters>
  <locations>
    <location type='data' path='/user/ambari-qa/falcon/'>
    </location>
    <location type='stats' path='/'>
    </location> 
    <location type='meta' path='/'>
    </location> 
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/> 
  <schema location='/none' provider='/none'/> 
  <properties>
    <property name='jobPriority' value='NORMAL'> 
    </property>
  </properties> 
</feed>

An I am sure the data path exist :

-rw-r--r--   3 hdfs      hdfs          0 2016-02-15 09:46 /user/ambari-qa/falcon/test                          
-rwxrwxrwx   3 ambari-qa hdfs          0 2016-02-15 09:33 /user/ambari-qa/falcon/testFeed

I don't understand why I need a time component ?

Re: Feed : Unable to resolve pattern for feedPath

bleonhardi — Mon, 15 Feb 2016 18:40:26 GMT

You have a retention period for your feed. Which means Falcon comes up and tries to delete all folders that belong to feed instances that are older than 5 minutes.

So he expects something like:

path='/inputfolder/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}'

So folders in HDFS that look like

/inputfolder/2016-01-01-09-15

etc. pp. And then he can delete all folders that are older than that.

It is all very nicely explained in the oozie documentation ( Falcon is build on oozie )

http://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a5._Dataset

Re: Feed : Unable to resolve pattern for feedPath

arthur_grevin — Fri, 19 Feb 2016 17:26:47 GMT

@Benjamin Leonhardi

I still have the same error even if when I change the data path :

org.apache.falcon.FalconException: Couldn't evict feed from fileSystem

And the feed is :

<feed xmlns='uri:falcon:feed:0.1' name='Depot'>
  <tags>projet=103,data=raw</tags>
  <frequency>hours(1)</frequency>
  <timezone>UTC</timezone>
  <clusters>
    <cluster name='primaryCluster' type='source'>
      <validity start='2016-02-18T10:11Z' end='2016-02-19T10:11Z'/>
      <retention limit='hours(4)' action='delete'/>
      <locations>
        <location type='data'>
        </location>
        <location type='stats'>
        </location>
        <location type='meta'>
        </location>
      </locations>
    </cluster>
  </clusters>
  <locations>
    <location type='data' path='/user/hdfs/falcon/DepotProject103/${YEAR}-${MONTH}-${DAY}-${HOUR}'>
    </location>
    <location type='stats' path='/'>
    </location>
    <location type='meta' path='/'>
    </location>
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/>
  <schema location='/none' provider='/none'/>
  <properties>
    <property name='jobPriority' value='NORMAL'>
    </property>
  </properties>
</feed>

I think I did it right this time ? Is there something I am missing again ?

Re: Feed : Unable to resolve pattern for feedPath

mrakki3110 — Wed, 08 Jun 2016 21:42:44 GMT

Whats the solution to this problem ??

Re: Feed : Unable to resolve pattern for feedPath

bvellanki — Wed, 08 Jun 2016 21:49:27 GMT

Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.

Re: Feed : Unable to resolve pattern for feedPath

mrakki3110 — Thu, 09 Jun 2016 14:34:10 GMT

@Balu : I changed the "stats" & "meta" location in Feed as you mentioned....that error got resolved & eviction got succeeded..thanks for the answer