Hi, During replication/retention Feed some files are replicated and deleted. Do you know how I can check what files are replicated and deleted except to make 'hdfs dfs -ls', so is there a log who lists all files impacted ? regards ... View more

Hi all, It seems that data path should like if frequency feed is "hours(2)" : /tmp/falcon/next-vers-current/${YEAR}/${MONTH}/${DAY}/${HOUR} My question is : all the paths need to be create before on primary and backup cluster ? /tmp/falcon/next-vers-current/2016/05/26/13/ /tmp/falcon/next-vers-current/2016/05/26/14/ /tmp/falcon/next-vers-current/2016/05/26/15/ ... View more

Hello, During feed replication (replication hdfs between clusterA and clusterB was done correctly), there's issue with FALCON_FEED_RETENTION concerning Path /tmp/falcon/next-vers-current, this path exists on clusterA and clusterB. Scheduled job : <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <feed name="next-vers-current" description="next-vers-current" xmlns="uri:falcon:feed:0.1"> <frequency>hours(6)</frequency> <timezone>UTC</timezone> <clusters> <cluster name="next-rec-cluster" type="source"> <validity start="2016-05-01T12:00Z" end="2016-05-27T23:00Z"/> <retention limit="days(2)" action="delete"/> <locations> <location type="data" path="/tmp/falcon/next-vers-current"/> </locations> </cluster> <cluster name="current-rec-cluster" type="target"> <validity start="2016-05-01T12:00Z" end="2016-05-27T23:00Z"/> <retention limit="days(2)" action="delete"/> <locations> <location type="data" path="/tmp/falcon/next-vers-current"/> </locations> </cluster> </clusters> <locations> <location type="data" path="/tmp/falcon/"/> <location type="stats" path="/none"/> <location type="meta" path="/none"/> </locations> <ACL owner="falcon" group="hadoop" permission="0755"/> <schema location="/none" provider="none"/> <properties><property name="queueName" value="oozie-launcher"/></properties> </feed> 2016-05-25 15:54:15,369 WARN JavaActionExecutor:523 - SERVER[clusterA] USER[falcon] GROUP[-] TOKEN[] APP[FALCON_FEED_RETENTION_next-vers-current] JOB[0000002-160525113126752-oozie-oozi-W] ACTION[0000002-160525113126752-oozie-oozi-W@eviction] Launcher exception: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem org.apache.oozie.action.hadoop.JavaMainException: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:59) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.JavaMain.main(JavaMain.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Caused by: org.apache.falcon.FalconException: Couldn't evict feed from fileSystem at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:306) at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:76) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:52) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.JavaMain.run(JavaMain.java:56) ... 15 more Caused by: java.io.IOException: Unable to resolve pattern for feedPath: /tmp/falcon/next-vers-current at org.apache.falcon.entity.FeedHelper.getFeedBasePath(FeedHelper.java:442) at org.apache.falcon.entity.FileSystemStorage.fileSystemEvictor(FileSystemStorage.java:331) at org.apache.falcon.entity.FileSystemStorage.evict(FileSystemStorage.java:300) ... 23 more ... View more