[RESOLVED] [FALCON] Replication/Retention Feed

[#1]

Hi,

During replication/retention Feed some files are replicated and deleted.

Do you know how I can check what files are replicated and deleted except to make 'hdfs dfs -ls', so is there a log who lists all files impacted ?

regards

[#2]

Hi @mayki wogno,

Can you give more details about your feed?

[#3]

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="next-vers-current" description="next-vers-current" xmlns="uri:falcon:feed:0.1">
 <frequency>hours(2)</frequency>
 <timezone>UTC</timezone>
 <late-arrival cut-off="hours(1)"/>

 <clusters>
  <cluster name="next-rec-cluster" type="source">
 <validity start="2016-05-27T12:00Z" end="2016-05-31T23:00Z"/>
 <retention limit="hours(6)" action="delete"/>
 </cluster>

 <cluster name="current-rec-cluster" type="target">
 <validity start="2016-05-01T13:00Z" end="2016-05-31T23:00Z"/>
 <retention limit="days(6)" action="delete"/>
 <locations>
 <location type="data" path="/tmp/falcon/next-vers-current/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
 </locations>
 </cluster>
 </clusters>

 <locations>
 <location type="data" path="/tmp/falcon/next-vers-current/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
 <location type="stats" path="/none"/>
 <location type="meta" path="/none"/>
 </locations>
 <ACL owner="falcon" group="hadoop" permission="0755"/>
 <schema location="/none" provider="none"/>
 <properties><property name="queueName" value="oozie-launcher"/></properties>
</feed>

[#4 · Accepted solution]

@mayki wogno

You can check staging location of the cluster.

In my case it is something like /apps/falcon/<clustername>/staging/workflows/feed/<feed-name>/logs

You can check your source and target cluster definition for staging location.

Something like below:

$ falcon entity -type cluster -definition -name backup

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<cluster name="backup" description="backup" colo="backup" xmlns="uri:falcon:cluster:0.1">
    <tags>EntityType=Cluster</tags>    
    <interfaces>        
    <interface type="readonly" endpoint="hdfs://nr21.hwxblr.com:50070" version="2.7.1"/>
    <interface type="write" endpoint="hdfs://nr21.hwxblr.com:8020" version="2.7.1"/>        
    <interface type="execute" endpoint="nr23.hwxblr.com:8050" version="2.7.1"/>        
    <interface type="workflow" endpoint="http://nr22.hwxblr.com:11000/oozie/" version="4.2.0"/>                    <interface type="messaging" endpoint="tcp://nr22.hwxblr.com:61616?daemon=true" version="5.1.6"/>          <interface type="registry" endpoint="thrift://nr22.hwxblr.com:9083" version="1.2.1"/>
</interfaces>    
<locations>        
<location name="staging" path="/apps/falcon/backup/staging"/>        
<location name="temp" path="/tmp"/>        
<location name="working" path="/apps/falcon/backup/working"/>    
</locations>    
<ACL owner="falcon" group="users" permission="0x755"/>
</cluster>

Do let us know if it was helpful.

RahulP

[#5]

Great Thanks.