Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Feed : Unable to resolve pattern for feedPath

Solved Go to solution
Highlighted

Feed : Unable to resolve pattern for feedPath

Hi,

I get this error when I run a Feed. My feed is running but when i check into the cluster I find this in the log :

org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
	at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306)
	at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
	... 15 more
Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /user/ambari-qa/falcon
	at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:435)
	at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331)
	at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300)
	... 23 more

I don't really know what does this mean and How do i fix this ?

Am I right when I expect to see the file of the dataPath I provide (/usr/ambari-qa/falcon) in my cluster ?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Feed : Unable to resolve pattern for feedPath

Rising Star

Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.

View solution in original post

7 REPLIES 7
Highlighted

Re: Feed : Unable to resolve pattern for feedPath

Can you add your feed xml? There should be a location something like this:

<locations> <location type="data" path="/mydata/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/> </locations>

Normally feeds are parametrized to correspond to the frequency of the feed.

So my question is, does the folder exist in HDFS? Do you have any retention policy? Eviction sounds like that and would not work in a folder without a time component. Easiest way would be to upload your feed xml.

Highlighted

Re: Feed : Unable to resolve pattern for feedPath

@Benjamin Leonhardi

Here is my feed xml

<feed xmlns='uri:falcon:feed:0.1' name='testFeed2'>
  <frequency>minutes(5)</frequency>
  <timezone>GMT+01:00</timezone> 
  <clusters>
    <cluster name='primaryCluster' type='source'> 
      <validity start='2016-02-15T08:41Z' end='2016-02-16T09:41Z'/>
      <retention limit='minutes(5)' action='delete'/>
      <locations>
        <location type='data'> 
        </location>
        <location type='stats'>
        </location>
        <location type='meta'> 
        </location>
      </locations>
    </cluster> 
  </clusters>
  <locations>
    <location type='data' path='/user/ambari-qa/falcon/'>
    </location>
    <location type='stats' path='/'>
    </location> 
    <location type='meta' path='/'>
    </location> 
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/> 
  <schema location='/none' provider='/none'/> 
  <properties>
    <property name='jobPriority' value='NORMAL'> 
    </property>
  </properties> 
</feed>

An I am sure the data path exist :

-rw-r--r--   3 hdfs      hdfs          0 2016-02-15 09:46 /user/ambari-qa/falcon/test                          
-rwxrwxrwx   3 ambari-qa hdfs          0 2016-02-15 09:33 /user/ambari-qa/falcon/testFeed   

I don't understand why I need a time component ?

Highlighted

Re: Feed : Unable to resolve pattern for feedPath

You have a retention period for your feed. Which means Falcon comes up and tries to delete all folders that belong to feed instances that are older than 5 minutes.

So he expects something like:

path='/inputfolder/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}'

So folders in HDFS that look like

/inputfolder/2016-01-01-09-15

etc. pp. And then he can delete all folders that are older than that.

It is all very nicely explained in the oozie documentation ( Falcon is build on oozie )

http://oozie.apache.org/docs/4.2.0/CoordinatorFunctionalSpec.html#a5._Dataset

Re: Feed : Unable to resolve pattern for feedPath

@Benjamin Leonhardi

I still have the same error even if when I change the data path :

org.apache.falcon.FalconException: Couldn't evict feed from fileSystem

And the feed is :

<feed xmlns='uri:falcon:feed:0.1' name='Depot'>
  <tags>projet=103,data=raw</tags>
  <frequency>hours(1)</frequency>
  <timezone>UTC</timezone>
  <clusters>
    <cluster name='primaryCluster' type='source'>
      <validity start='2016-02-18T10:11Z' end='2016-02-19T10:11Z'/>
      <retention limit='hours(4)' action='delete'/>
      <locations>
        <location type='data'>
        </location>
        <location type='stats'>
        </location>
        <location type='meta'>
        </location>
      </locations>
    </cluster>
  </clusters>
  <locations>
    <location type='data' path='/user/hdfs/falcon/DepotProject103/${YEAR}-${MONTH}-${DAY}-${HOUR}'>
    </location>
    <location type='stats' path='/'>
    </location>
    <location type='meta' path='/'>
    </location>
  </locations>
  <ACL owner='ambari-qa' group='users' permission='0x755'/>
  <schema location='/none' provider='/none'/>
  <properties>
    <property name='jobPriority' value='NORMAL'>
    </property>
  </properties>
</feed>

I think I did it right this time ? Is there something I am missing again ?

Highlighted

Re: Feed : Unable to resolve pattern for feedPath

Explorer

Whats the solution to this problem ??

Highlighted

Re: Feed : Unable to resolve pattern for feedPath

Rising Star

Can you please change the "stats" and "meta" location in Feed to /tmp/${YEAR}-${MONTH}-${DAY}-${HOUR} and try again? If updating stats and meta locations will ensure eviction succeeds, then this is an improvement we will have to make in Falcon.

View solution in original post

Highlighted

Re: Feed : Unable to resolve pattern for feedPath

Explorer

@Balu : I changed the "stats" & "meta" location in Feed as you mentioned....that error got resolved & eviction got succeeded..thanks for the answer

Don't have an account?
Coming from Hortonworks? Activate your account here