Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Feed Replication Only after Process Completion

SOLVED Go to solution
Highlighted

Feed Replication Only after Process Completion

Contributor

Scenario: we have data that is ingested to the cluster via a Falcon process. It leverages a Falcon feed as an output to replicate the ingested data to a backup cluster. We'd like the feed to not replicate until the process has completed. We currently use a delay to semi-accomplish this, but it's not perfect.

Question: How can we (if at all) tell Falcon to wait until the process has completed to begin replication? Currently we have process.xml code as below, which tells the Feed to start "now" but with a delay of 2 hours (this delay being specified in the feed.xml).

To be clear, we have replication working and all that just fine -- we're just after a more elegant way for Falcon to only replicate after a process has confirmed completed. Is there a way?

Process

<outputs>
    <output name="hdp0001-my-feed" feed="hdp0001-my-feed" instance="now(0,0)"/>
</outputs>

Feed

<cluster name="primary-cluster" type="source" delay="hours(1)">
    <validity start="2015-12-04T09:30Z" end="2099-12-31T23:59Z"/>
    <retention limit="months(9999)" action="archive"/>
    <locations>
        <location type="data" path="/hdfs/data/path/to/my_table/"/>
    </locations>
</cluster>
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Feed Replication Only after Process Completion

Rising Star

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>
5 REPLIES 5

Re: Feed Replication Only after Process Completion

Rising Star

In the feed entity specification, http://falcon.apache.org/EntitySpecification.html#... please look for availability flag. A feed is considered available for downstream consumption, replication etc when the availabilityFlag file is created. You can make your process create the availabilityFlag as last task. Please let me know if this works.

    <availabilityFlag>_SUCCESS</availabilityFlag>

Re: Feed Replication Only after Process Completion

Contributor

Balu, thanks for that answer! We knew of this tag, but weren't sure how to truly use it. I'm testing this now and will let you know asap! Thank you kindly.

Re: Feed Replication Only after Process Completion

Contributor

Hey Balu, that worked. Perfectly, actually!

The only thing left I was curious about... is there a Falcon-based way to remove the _SUCCESS file after replication has completed? I know how we can do it after some time or after creating it in the process (and waiting a few minutes perhaps), but if there's a Falcon method or tag I'd love to leverage that.

Please let me know! And thanks for all the great help.

Re: Feed Replication Only after Process Completion

Rising Star

Unfortunately, there is no Falcon way to do this. Sorry.

Re: Feed Replication Only after Process Completion

Rising Star

You are very welcome, Happy to be of help.