Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

Solved Go to solution
Highlighted

Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

When using the Falcon "Mirror Recipe", what happens if an instance is still running when the next is scheduled to start?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

If you are asking about using recipes for mirroring, if you look at hdfs-replication-template.xml or hive-replication-template.xml has below properties set to ensure only one instance runs.

<parallel>1</parallel>
<!-- Dir replication needs to run only once to catch up -->     
<order>LAST_ONLY</order>

--------------------------------------------------------------------------------------------------------------

In general, in Falcon this can be controlled using <parallel> and <order> properties in the entity xml. Parallel decides the concurrent replication instances that can run at any given time and order decides the execution order for replication instances like FIFO, LIFO and LAST_ONLY. For replication job needs to run only once to catch up. Setting below configs in entity xml will ensure this:

<parallel>1</parallel>

<order>LAST_ONLY</order>

Parallel maps to concurrency and order maps to execution in Oozie. Please refer Oozie doc for more details.

View solution in original post

5 REPLIES 5
Highlighted

Re: Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

Rising Star

The Falcon "Mirror" Recipe has "parallelism" set to 1 since replication needs to run only once to catch up. The next instance will wait for the current instance to complete.

Highlighted

Re: Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

Good answer. Somwa's was more complete with a description of how it works.

Highlighted

Re: Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

If you are asking about using recipes for mirroring, if you look at hdfs-replication-template.xml or hive-replication-template.xml has below properties set to ensure only one instance runs.

<parallel>1</parallel>
<!-- Dir replication needs to run only once to catch up -->     
<order>LAST_ONLY</order>

--------------------------------------------------------------------------------------------------------------

In general, in Falcon this can be controlled using <parallel> and <order> properties in the entity xml. Parallel decides the concurrent replication instances that can run at any given time and order decides the execution order for replication instances like FIFO, LIFO and LAST_ONLY. For replication job needs to run only once to catch up. Setting below configs in entity xml will ensure this:

<parallel>1</parallel>

<order>LAST_ONLY</order>

Parallel maps to concurrency and order maps to execution in Oozie. Please refer Oozie doc for more details.

View solution in original post

Highlighted

Re: Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

@Sowmya Ramesh - Good answer! Since this was specific to "recipes", can you move the recipe part of the question to the top of your answer and keep the further detail on "parallel" and "order" after that. Then I can accept.

Highlighted

Re: Falcon mirror recipe: What happens if an instance is still running when the next is scheduled to start?

@Sean Roberts: Done!

Don't have an account?
Coming from Hortonworks? Activate your account here