Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Feed : Unable to resolve pattern for feedPath

avatar
Rising Star

Hi,

I get this error when I run a Feed. My feed is running but when i check into the cluster I find this in the log :

org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
	at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306)
	at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
	... 15 more
Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /user/ambari-qa/falcon
	at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:435)
	at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331)
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300)
	... 23 more

I don't really know what does this mean and How do i fix this ?

Am I right when I expect to see the file of the dataPath I provide (/usr/ambari-qa/falcon) in my cluster ?

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.

View solution in original post

7 REPLIES 7

avatar
Master Guru

Can you add your feed xml? There should be a location something like this:

<locations> <location type="data" path="/mydata/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> </locations>

Normally feeds are parametrized to correspond to the frequency of the feed.

So my question is, does the folder exist in HDFS? Do you have any retention policy? Eviction sounds like that and would not work in a folder without a time component. Easiest way would be to upload your feed xml.

avatar
Rising Star

@Benjamin Leonhardi

Here is my feed xml

<feed xmlns='uri:falcon:feed:0.1' name='testFeed2'>
  <frequency>minutes(5)</frequency>
  <timezone>GMT+01:00</timezone> 
  <clusters>
    <cluster name='primaryCluster' type='source'> 
      <validity start='2016-02-15T08:41Z' end='2016-02-16T09:41Z'/>
      <retention limit='minutes(5)' action='delete'/>
      <locations>
        <location type='data'> 
        </location>
        <location type='stats'>
        </location>
        <location type='meta'> 
        </location>
      </locations>
    </cluster> 
  </clusters>
  <locations>
    <location type='data' path='/user/ambari-qa/falcon/'>
    </location>
    <location type='stats' path='/'>
    </location> 
    <location type='meta' path='/'>
    </location> 
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/> 
  <schema location='/none' provider='/none'/> 
  <properties>
    <property name='jobPriority' value='NORMAL'> 
    </property>
  </properties> 
</feed>

An I am sure the data path exist :

-rw-r--r--   3 hdfs      hdfs          0 2016-02-15 09:46 /user/ambari-qa/falcon/test                          
-rwxrwxrwx   3 ambari-qa hdfs          0 2016-02-15 09:33 /user/ambari-qa/falcon/testFeed   

I don't understand why I need a time component ?

avatar
Master Guru

You have a retention period for your feed. Which means Falcon comes up and tries to delete all folders that belong to feed instances that are older than 5 minutes.

So he expects something like:

path='/inputfolder/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}'

So folders in HDFS that look like

/inputfolder/2016-01-01-09-15

etc. pp. And then he can delete all folders that are older than that.

It is all very nicely explained in the oozie documentation ( Falcon is build on oozie )

http://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a5._Dataset

avatar
Rising Star

@Benjamin Leonhardi

I still have the same error even if when I change the data path :

org.apache.falcon.FalconException: Couldn't evict feed from fileSystem

And the feed is :

<feed xmlns='uri:falcon:feed:0.1' name='Depot'>
  <tags>projet=103,data=raw</tags>
  <frequency>hours(1)</frequency>
  <timezone>UTC</timezone>
  <clusters>
    <cluster name='primaryCluster' type='source'>
      <validity start='2016-02-18T10:11Z' end='2016-02-19T10:11Z'/>
      <retention limit='hours(4)' action='delete'/>
      <locations>
        <location type='data'>
        </location>
        <location type='stats'>
        </location>
        <location type='meta'>
        </location>
      </locations>
    </cluster>
  </clusters>
  <locations>
    <location type='data' path='/user/hdfs/falcon/DepotProject103/${YEAR}-${MONTH}-${DAY}-${HOUR}'>
    </location>
    <location type='stats' path='/'>
    </location>
    <location type='meta' path='/'>
    </location>
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/>
  <schema location='/none' provider='/none'/>
  <properties>
    <property name='jobPriority' value='NORMAL'>
    </property>
  </properties>
</feed>

I think I did it right this time ? Is there something I am missing again ?

avatar
Contributor

Whats the solution to this problem ??

avatar
Expert Contributor

Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.

avatar
Contributor

@Balu : I changed the "stats" & "meta" location in Feed as you mentioned....that error got resolved & eviction got succeeded..thanks for the answer