Feed : Unable to resolve pattern for feedPath

[#1]

Hi,

I get this error when I run a Feed. My feed is running but when i check into the cluster I find this in the log :

org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
	at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306)
	at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
	... 15 more
Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /user/ambari-qa/falcon
	at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:435)
	at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331)
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300)
	... 23 more

I don't really know what does this mean and How do i fix this ?

Am I right when I expect to see the file of the dataPath I provide (/usr/ambari-qa/falcon) in my cluster ?

[#2]

Can you add your feed xml? There should be a location something like this:

<locations> <location type="data" path="/mydata/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> </locations>

Normally feeds are parametrized to correspond to the frequency of the feed.

So my question is, does the folder exist in HDFS? Do you have any retention policy? Eviction sounds like that and would not work in a folder without a time component. Easiest way would be to upload your feed xml.

[#3]

@Benjamin Leonhardi

Here is my feed xml

<feed xmlns='uri:falcon:feed:0.1' name='testFeed2'>
  <frequency>minutes(5)</frequency>
  <timezone>GMT+01:00</timezone> 
  <clusters>
    <cluster name='primaryCluster' type='source'> 
      <validity start='2016-02-15T08:41Z' end='2016-02-16T09:41Z'/>
      <retention limit='minutes(5)' action='delete'/>
      <locations>
        <location type='data'> 
        </location>
        <location type='stats'>
        </location>
        <location type='meta'> 
        </location>
      </locations>
    </cluster> 
  </clusters>
  <locations>
    <location type='data' path='/user/ambari-qa/falcon/'>
    </location>
    <location type='stats' path='/'>
    </location> 
    <location type='meta' path='/'>
    </location> 
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/> 
  <schema location='/none' provider='/none'/> 
  <properties>
    <property name='jobPriority' value='NORMAL'> 
    </property>
  </properties> 
</feed>

An I am sure the data path exist :

-rw-r--r--   3 hdfs      hdfs          0 2016-02-15 09:46 /user/ambari-qa/falcon/test                          
-rwxrwxrwx   3 ambari-qa hdfs          0 2016-02-15 09:33 /user/ambari-qa/falcon/testFeed   

I don't understand why I need a time component ?

[#4]

You have a retention period for your feed. Which means Falcon comes up and tries to delete all folders that belong to feed instances that are older than 5 minutes.

So he expects something like:

path='/inputfolder/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}'

So folders in HDFS that look like

/inputfolder/2016-01-01-09-15

etc. pp. And then he can delete all folders that are older than that.

It is all very nicely explained in the oozie documentation ( Falcon is build on oozie )

http://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a5._Dataset

[#5]

@Benjamin Leonhardi

I still have the same error even if when I change the data path :

org.apache.falcon.FalconException: Couldn't evict feed from fileSystem

And the feed is :

<feed xmlns='uri:falcon:feed:0.1' name='Depot'>
  <tags>projet=103,data=raw</tags>
  <frequency>hours(1)</frequency>
  <timezone>UTC</timezone>
  <clusters>
    <cluster name='primaryCluster' type='source'>
      <validity start='2016-02-18T10:11Z' end='2016-02-19T10:11Z'/>
      <retention limit='hours(4)' action='delete'/>
      <locations>
        <location type='data'>
        </location>
        <location type='stats'>
        </location>
        <location type='meta'>
        </location>
      </locations>
    </cluster>
  </clusters>
  <locations>
    <location type='data' path='/user/hdfs/falcon/DepotProject103/${YEAR}-${MONTH}-${DAY}-${HOUR}'>
    </location>
    <location type='stats' path='/'>
    </location>
    <location type='meta' path='/'>
    </location>
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/>
  <schema location='/none' provider='/none'/>
  <properties>
    <property name='jobPriority' value='NORMAL'>
    </property>
  </properties>
</feed>

I think I did it right this time ? Is there something I am missing again ?

[#6]

Whats the solution to this problem ??

[#7 · Accepted solution]

Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.

[#8]

@Balu : I changed the "stats" & "meta" location in Feed as you mentioned....that error got resolved & eviction got succeeded..thanks for the answer