Feed Replication Only after Process Completion

[#1]

Scenario: we have data that is ingested to the cluster via a Falcon process. It leverages a Falcon feed as an output to replicate the ingested data to a backup cluster. We'd like the feed to not replicate until the process has completed. We currently use a delay to semi-accomplish this, but it's not perfect.

Question: How can we (if at all) tell Falcon to wait until the process has completed to begin replication? Currently we have process.xml code as below, which tells the Feed to start "now" but with a delay of 2 hours (this delay being specified in the feed.xml).

To be clear, we have replication working and all that just fine -- we're just after a more elegant way for Falcon to only replicate after a process has confirmed completed. Is there a way?

Process

<outputs>
    <output name="hdp0001-my-feed" feed="hdp0001-my-feed" instance="now(0,0)"/>
</outputs>

Feed

<cluster name="primary-cluster" type="source" delay="hours(1)">
    <validity start="2015-12-04T09:30Z" end="2099-12-31T23:59Z"/>
    <retention limit="months(9999)" action="archive"/>
    <locations>
        <location type="data" path="/hdfs/data/path/to/my_table/"/>
    </locations>
</cluster>

[#2 · Accepted solution]

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>

[#3]

Balu, thanks for that answer! We knew of this tag, but weren't sure how to truly use it. I'm testing this now and will let you know asap! Thank you kindly.

[#4]

Hey Balu, that worked. Perfectly, actually!

The only thing left I was curious about... is there a Falcon-based way to remove the _SUCCESS file after replication has completed? I know how we can do it after some time or after creating it in the process (and waiting a few minutes perhaps), but if there's a Falcon method or tag I'd love to leverage that.

Please let me know! And thanks for all the great help.

[#5]

Unfortunately, there is no Falcon way to do this. Sorry.

[#6]

You are very welcome, Happy to be of help.