Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Feed Replication Only after Process Completion

avatar
Rising Star

Scenario: we have data that is ingested to the cluster via a Falcon process. It leverages a Falcon feed as an output to replicate the ingested data to a backup cluster. We'd like the feed to not replicate until the process has completed. We currently use a delay to semi-accomplish this, but it's not perfect.

Question: How can we (if at all) tell Falcon to wait until the process has completed to begin replication? Currently we have process.xml code as below, which tells the Feed to start "now" but with a delay of 2 hours (this delay being specified in the feed.xml).

To be clear, we have replication working and all that just fine -- we're just after a more elegant way for Falcon to only replicate after a process has confirmed completed. Is there a way?

Process

<outputs>
    <output name="hdp0001-my-feed" feed="hdp0001-my-feed" instance="now(0,0)"/>
</outputs>

Feed

<cluster name="primary-cluster" type="source" delay="hours(1)">
    <validity start="2015-12-04T09:30Z" end="2099-12-31T23:59Z"/>
    <retention limit="months(9999)" action="archive"/>
    <locations>
        <location type="data" path="/hdfs/data/path/to/my_table/"/>
    </locations>
</cluster>
1 ACCEPTED SOLUTION

avatar
Expert Contributor

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>

View solution in original post

5 REPLIES 5

avatar
Expert Contributor

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>

avatar
Rising Star

Balu, thanks for that answer! We knew of this tag, but weren't sure how to truly use it. I'm testing this now and will let you know asap! Thank you kindly.

avatar
Rising Star

Hey Balu, that worked. Perfectly, actually!

The only thing left I was curious about... is there a Falcon-based way to remove the _SUCCESS file after replication has completed? I know how we can do it after some time or after creating it in the process (and waiting a few minutes perhaps), but if there's a Falcon method or tag I'd love to leverage that.

Please let me know! And thanks for all the great help.

avatar
Expert Contributor

Unfortunately, there is no Falcon way to do this. Sorry.

avatar
Expert Contributor

You are very welcome, Happy to be of help.