Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Yarn preemption and Oozie

avatar
Master Guru

I wanted to know if people around here have experience with oozie and yarn preemption.

I think I remember that the two do not work well with each other.

I.e. lets assume we have the launcher Application Master, the launcher map task and the Task ( perhaps pig ) Application Master and the Pig tasks

So there are 4 possibilities:

A) Pig Container is killed

Should be fine, pig will reschedule it through the application master

B) Pig Application Master is killed

Should be rare since preemption kills Application Masters only as a last resort. I assume the oozie launcher would fail but that there is a retry parameter in oozie?

C) Oozie launcher Map is killed

Suddenly the pig task is orphaned. Will oozie application master restart the map? Will the map reconnect to the pig task? Or will it start a second one?

D) Oozie launcher AM is killed

Similar to C) but will oozie server restart the task or will it be shown as killed

I also remember an engagement where they had orphaned tasks because of oozie and preemption anybody seen someting like that?

Thanks a lot.

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
4 REPLIES 4

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Mentor

I like the Oozie dedicated queue idea! @Predrag Minovic

avatar
Master Guru

Uber task sounds interesting I need to check this out. Also having a dedicated sub queue for the oozie actions is most likely the way for me to go because I have one type of action that doesn't take a kill nicely. ( It would result in duplicated data ).

I just remember the summit last year when someone mentioned that they had some configuration where preemption resulted in essentially endlessly running tasks. I cannot remember the constellation when this happens.

avatar
New Contributor

I have seen so.ething similar caused by YARN-2964