[RESOLVED] [FALCON] error during feed retention delete

maykiwogno — Sun, 18 Aug 2019 11:07:38 GMT

Hello,

During feed replication (replication hdfs between clusterA and clusterB was done correctly), there's issue with FALCON_FEED_RETENTION concerning Path /tmp/falcon/next-vers-current, this path exists on clusterA and clusterB.

Scheduled job :

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="next-vers-current" description="next-vers-current" xmlns="uri:falcon:feed:0.1">
 <frequency>hours(6)</frequency>
 <timezone>UTC</timezone>
 <clusters>
 <cluster name="next-rec-cluster" type="source">
 <validity start="2016-05-01T12:00Z" end="2016-05-27T23:00Z"/>
 <retention limit="days(2)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/next-vers-current"/>
 </locations>
 </cluster>
 <cluster name="current-rec-cluster" type="target">
 <validity start="2016-05-01T12:00Z" end="2016-05-27T23:00Z"/>
 <retention limit="days(2)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/next-vers-current"/>
 </locations>
 </cluster>
 </clusters>
 <locations>
 <location type="data" path="/tmp/falcon/"/>
 <location type="stats" path="/none"/>
 <location type="meta" path="/none"/>
 </locations>
 <ACL owner="falcon" group="hadoop" permission="0755"/>
 <schema location="/none" provider="none"/>
 <properties><property name="queueName" value="oozie-launcher"/></properties>
</feed>

2016-05-25 15:54:15,369  WARN JavaActionExecutor:523 - SERVER[clusterA] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_FEED_RETENTION_next-vers-current] JOB[0000002-160525113126752-oozie-oozi-W] ACTION[0000002-160525113126752-oozie-oozi-W@eviction] Launcher exception: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
    at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem
    at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306)
    at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56)
    ... 15 more
Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /tmp/falcon/next-vers-current
    at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:442)
    at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331)
    at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300)
    ... 23 more

Re: [RESOLVED] [FALCON] error during feed retention delete

pminovic — Sat, 28 May 2016 06:48:32 GMT

Try to set you location to:

<locationtype="data"path="/tmp/falcon/next-vers-current/${YEAR}/${MONTH}/${DAY}/${HOUR}"/>

From the feed specification page: The granularity of date pattern in the path should be at least that of a frequency of a feed.

Re: [RESOLVED] [FALCON] error during feed retention delete

pminovic — Sun, 29 May 2016 20:43:34 GMT

Hi @mayki wogno, I see that you marked your question as "Resolved". If my answer below helped you can you please accept it. If you resolved your issue by other means please publish them, and we'll accept your answer. Instead of marking questions as "resolved" we consider them resolved if they are accepted. Tnx!

question Re: [RESOLVED] [FALCON] error during feed retention delete in Archives of Support Questions (Read Only)

[RESOLVED] [FALCON] error during feed retention delete

Re: [RESOLVED] [FALCON] error during feed retention delete

Re: [RESOLVED] [FALCON] error during feed retention delete