Yarn preemption and Oozie

[#1]

I wanted to know if people around here have experience with oozie and yarn preemption.

I think I remember that the two do not work well with each other.

I.e. lets assume we have the launcher Application Master, the launcher map task and the Task ( perhaps pig ) Application Master and the Pig tasks

So there are 4 possibilities:

A) Pig Container is killed

Should be fine, pig will reschedule it through the application master

B) Pig Application Master is killed

Should be rare since preemption kills Application Masters only as a last resort. I assume the oozie launcher would fail but that there is a retry parameter in oozie?

C) Oozie launcher Map is killed

Suddenly the pig task is orphaned. Will oozie application master restart the map? Will the map reconnect to the pig task? Or will it start a second one?

D) Oozie launcher AM is killed

Similar to C) but will oozie server restart the task or will it be shown as killed

I also remember an engagement where they had orphaned tasks because of oozie and preemption anybody seen someting like that?

Thanks a lot.

[#2 · Accepted solution]

Hi Ben, I'm not an Oozie expert buy you look lonely on this page, nobody to help you 🙂 So, the simplest solution is to have a special queue for Oozie apps and set maximum-capacity=capacity and preemption will not happen even if enabled. But you can say I'm avoiding the question 🙂 Otherwise, I think in all cases (A) - (D), if preemption happens respective tasks will be retried, but then we can have some orphaned tasks. So, we'd like to avoid in particular A, B and C. A and C are AMs which can be controlled by maximum-am-resource-percent per queue, and the Scheduler is not supposed to kill AMs I guess. Then only B remains exposed, and by doing some search I found that we can run Oozie action launcher in uber mode so that A and B run together, the property is called oozie.action.launcher.mapreduce.job.ubertask.enable which by default is false (in spite of this jira) and can be set per action. HTH.

[#3]

I like the Oozie dedicated queue idea! @Predrag Minovic

[#4]

Uber task sounds interesting I need to check this out. Also having a dedicated sub queue for the oozie actions is most likely the way for me to go because I have one type of action that doesn't take a kill nicely. ( It would result in duplicated data ).

I just remember the summit last year when someone mentioned that they had some configuration where preemption resulted in essentially endlessly running tasks. I cannot remember the constellation when this happens.

[#5]

I have seen so.ething similar caused by YARN-2964