[RESOLVED] [FALCON] error during feed retention delete

[#1]

Hello,

During feed replication (replication hdfs between clusterA and clusterB was done correctly), there's issue with FALCON_FEED_RETENTION concerning Path /tmp/falcon/next-vers-current, this path exists on clusterA and clusterB.

Scheduled job :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="next-vers-current" description="next-vers-current" xmlns="uri:falcon:feed:0.1">
 <frequency>hours(6)</frequency>
 <timezone>UTC</timezone>
 <clusters>
 <cluster name="next-rec-cluster" type="source">
 <validity start="2016-05-01T12:00Z" end="2016-05-27T23:00Z"/>
 <retention limit="days(2)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/next-vers-current"/>
 </locations>
 </cluster>
 <cluster name="current-rec-cluster" type="target">
 <validity start="2016-05-01T12:00Z" end="2016-05-27T23:00Z"/>
 <retention limit="days(2)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/next-vers-current"/>
 </locations>
 </cluster>
 </clusters>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 <location type="stats" path="/none"/>
 <location type="meta" path="/none"/>
 </locations>
 <ACL owner="falcon" group="hadoop" permission="0755"/>
 <schema location="/none" provider="none"/>
 <properties><property name="queueName" value="oozie-launcher"/></properties>
</feed>

2016-05-25 15:54:15,369  WARN JavaActionExecutor:523 - SERVER[clusterA] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_FEED_RETENTION_next-vers-current] JOB[0000002-160525113126752-oozie-oozi-W] ACTION[0000002-160525113126752-oozie-oozi-W@eviction] Launcher exception: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
    at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306)
    at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
    ... 15 more
Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /tmp/falcon/next-vers-current
    at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:442)
    at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331)
    at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300)
    ... 23 more

[#2 · Accepted solution]

Try to set you location to:

<locationtype="data"path="/tmp/falcon/next-vers-current/${YEAR}/${MONTH}/${DAY}/${HOUR}"/>

From the feed specification page: The granularity of date pattern in the path should be at least that of a frequency of a feed.

[#3]

Hi @mayki wogno, I see that you marked your question as "Resolved". If my answer below helped you can you please accept it. If you resolved your issue by other means please publish them, and we'll accept your answer. Instead of marking questions as "resolved" we consider them resolved if they are accepted. Tnx!