Support Questions
Find answers, ask questions, and share your expertise

Oozie coordinator action execution order

Highlighted

Oozie coordinator action execution order

New Contributor

Hi Everybody,

As described here, there exist 4 Oozie's coordinator action execution order types (FIFO, LIFO, LAST_ONLY, NONE). We would like to use the "NONE" type, but so far our tests show that Oozie does not behave as expected.

As far as the documentation says, execution = "NONE" means that:

Similar to LAST_ONLY except all older materializations are skipped. When NONE is set, an action that is WAITING or READY will be SKIPPED when the current time is more than a certain configured number of minutes (tolerance, default 1 minute) past the action's nominal time.

Our TEST 1 was as such: coordinator.xml:

<coordinator-app
    name="test_coordinator"
    frequency="5" start="${coordinatorStart}" end="${coordinatorEnd}"
    timezone="Europe/Copenhagen" xmlns="uri:oozie:coordinator:0.4">
    <controls>
        <execution>NONE</execution>
 <throttle>1</throttle> 
    </controls>
    <action>
        <workflow>
            <app-path>${workflowAppUri}</app-path>
        </workflow>
    </action>
</coordinator-app>

This resulted in an factual execution order which went like this:

Frequency = 5 minutes

  • Action 50 started to run at nominal time 13:00 and finished at 13:22.
  • Actions 49 (13:05), 48 (13:10) and 47 (13:15) should have been put in SKIPPED state. In fact, they went into a READY state for a while.
  • Action 46 ,which was scheduled for nominal time 13:20, went in catch-up mode, and started to run right after 50 was finished (at 13:22, you can't see this in the pic ;-) )

Thus, it did correctly run ONLY THE LAST action (action 46) but it did not set status SKIPPED to 49, 48 and 47. They remained in READY state. Eventually, after the coordinator reached its end time, all the actions that were remained in READY status started to execute in a LIFO mode (see red marker in the picture, there were more not showed in the pic).... which is definitively NOT what we wanted!!! We wanted them DEAD

Thus 2 questions:

  1. Why are older action materializations being put into READY instead of SKIPPED ?
  2. and Why are they run in LIFO mode?

Thank you in advance if you have any idea of what's going on!

8702-capture.jpg

ADDITIONAL INFO:

We run HDP 2.3 and Oozie 4.2.0.

In TEST 2 we also tried to add a

<timeout>5</timeout>

but it did not help, Older actions materialization went into READY state, not SKIPPED.