Support Questions
Find answers, ask questions, and share your expertise

Feed Replication Only after Process Completion

Contributor

Scenario: we have data that is ingested to the cluster via a Falcon process. It leverages a Falcon feed as an output to replicate the ingested data to a backup cluster. We'd like the feed to not replicate until the process has completed. We currently use a delay to semi-accomplish this, but it's not perfect.

Question: How can we (if at all) tell Falcon to wait until the process has completed to begin replication? Currently we have process.xml code as below, which tells the Feed to start "now" but with a delay of 2 hours (this delay being specified in the feed.xml).

To be clear, we have replication working and all that just fine -- we're just after a more elegant way for Falcon to only replicate after a process has confirmed completed. Is there a way?

Process

<outputs>
    <output name="hdp0001-my-feed" feed="hdp0001-my-feed" instance="now(0,0)"/>
</outputs>

Feed

<cluster name="primary-cluster" type="source" delay="hours(1)">
    <validity start="2015-12-04T09:30Z" end="2099-12-31T23:59Z"/>
    <retention limit="months(9999)" action="archive"/>
    <locations>
        <location type="data" path="/hdfs/data/path/to/my_table/"/>
    </locations>
</cluster>
1 ACCEPTED SOLUTION

Rising Star

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>

View solution in original post

5 REPLIES 5

Rising Star

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>

Contributor

Balu, thanks for that answer! We knew of this tag, but weren't sure how to truly use it. I'm testing this now and will let you know asap! Thank you kindly.

Contributor

Hey Balu, that worked. Perfectly, actually!

The only thing left I was curious about... is there a Falcon-based way to remove the _SUCCESS file after replication has completed? I know how we can do it after some time or after creating it in the process (and waiting a few minutes perhaps), but if there's a Falcon method or tag I'd love to leverage that.

Please let me know! And thanks for all the great help.

Rising Star

Unfortunately, there is no Falcon way to do this. Sorry.

Rising Star

You are very welcome, Happy to be of help.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.