question Re: Feed Replication Only after Process Completion in Support Questions

Feed Replication Only after Process Completion

landon_t_robins — Tue, 22 Dec 2015 04:08:33 GMT

Scenario: we have data that is ingested to the cluster via a Falcon process. It leverages a Falcon feed as an output to replicate the ingested data to a backup cluster. We'd like the feed to not replicate until the process has completed. We currently use a delay to semi-accomplish this, but it's not perfect.

Question: How can we (if at all) tell Falcon to wait until the process has completed to begin replication? Currently we have process.xml code as below, which tells the Feed to start "now" but with a delay of 2 hours (this delay being specified in the feed.xml).

To be clear, we have replication working and all that just fine -- we're just after a more elegant way for Falcon to only replicate after a process has confirmed completed. Is there a way?

Process

<outputs>
    <output name="hdp0001-my-feed" feed="hdp0001-my-feed" instance="now(0,0)"/>
</outputs>

Feed

<cluster name="primary-cluster" type="source" delay="hours(1)">
    <validity start="2015-12-04T09:30Z" end="2099-12-31T23:59Z"/>
    <retention limit="months(9999)" action="archive"/>
    <locations>
        <location type="data" path="/hdfs/data/path/to/my_table/"/>
    </locations>
</cluster>

Re: Feed Replication Only after Process Completion

bvellanki — Tue, 22 Dec 2015 05:05:47 GMT

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>

Re: Feed Replication Only after Process Completion

landon_t_robins — Wed, 23 Dec 2015 03:58:52 GMT

Balu, thanks for that answer! We knew of this tag, but weren't sure how to truly use it. I'm testing this now and will let you know asap! Thank you kindly.

Re: Feed Replication Only after Process Completion

landon_t_robins — Wed, 23 Dec 2015 04:32:27 GMT

Hey Balu, that worked. Perfectly, actually!

The only thing left I was curious about... is there a Falcon-based way to remove the _SUCCESS file after replication has completed? I know how we can do it after some time or after creating it in the process (and waiting a few minutes perhaps), but if there's a Falcon method or tag I'd love to leverage that.

Please let me know! And thanks for all the great help.

Re: Feed Replication Only after Process Completion

bvellanki — Wed, 23 Dec 2015 05:46:15 GMT

Unfortunately, there is no Falcon way to do this. Sorry.

Re: Feed Replication Only after Process Completion

bvellanki — Wed, 23 Dec 2015 05:46:43 GMT

You are very welcome, Happy to be of help.