Support Questions

Find answers, ask questions, and share your expertise

Why might Oozie use coordinator start date (instead of the coordinator nominal date) for the special variables ${YEAR}, ${MONTH}, and ${DAY}?

avatar
New Contributor

Everything that I have read suggests that ${YEAR}, ${MONTH}, and ${DAY} can be used to specify the uri-template for date-based Oozie coordinator datasets (see, e.g., [1], [2]). However, after starting my coordinator via a bundle (see below) on '2017-08-26', Oozie continues to look in the directory '2017-08/26' for the presence of my trigger file (which has existed since the 26th). A StackOverflow post [3] was able to resolve a similar issue by replacing ${YEAR} with ${coord:formatTime(coord:nominalTime(),'yyyy')}, but this seems to break with the simplicity of using Oozie's EL Time Constants. Am I missing something regarding the proper usage of ${YEAR}, ${MONTH}, and ${DAY} for my coordinator's uri-template?

Thanks!

- Matt

Links:

[1] Oozie - . (2017). Oozie.apache.org. Retrieved 28 August 2017, from https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.html#a5.1._Synchronous_Datasets

[2] Wu, Jui-Chung. (2017). GitHub: YahooArchive/oozie. Retrieved 28 August 2017, from https://github.com/YahooArchive/oozie/wiki/Oozie-Coord-Use-Cases

[3] spijs. (2017). How to set current month, day or year as workflow parameter in a coordinator (on Hue). Stackoverflow.com. Retrieved 28 August 2017, from https://stackoverflow.com/questions/41744794/how-to-set-current-month-day-or-year-as-workflow-parame...

Oozie Version:

Oozie server build version: 4.2.0.2.5.0.0-1245

Bundle Source:

<bundle-app
    xmlns="uri:oozie:bundle:0.2" name="run_all_my_coordinators">
    <controls>
        <kick-off-time>${start_date}</kick-off-time>
    </controls>
    <coordinator name="notify_oozie_start">
        <app-path>${oozieProjectRoot}/notify_oozie_start</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_bundle_datatype}</value>
            </property>
        </configuration>
    </coordinator>


...
    
    <coordinator name="my_coordinator">
        <app-path>${coord_path}</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_datatype_name}</value>
            </property>
            <property>
                <name>datatype_dir</name>
                <value>${my_datatype_dir}</value>
            </property>
        </configuration>
    </coordinator>


...


    <coordinator name="notify_oozie_end">
        <app-path>${oozieProjectRoot}/notify_oozie_end</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_bundle_datatype}</value>
            </property>
        </configuration>
    </coordinator>
</bundle-app>

Coordinator Source:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<coordinator-app xmlns="uri:oozie:coordinator:0.5"
    name="my_coordinator"
    start="${start_date}"
    end="${end_date}"
    timezone="GMT-05:00"
    frequency="${coord:days(1)}">
    <controls>
        <concurrency>1</concurrency>
    </controls>
    <datasets>
        <dataset frequency="${coord:days(1)}"
            initial-instance="${initial_instance}"
            name="trigger_dataset_name"
            timezone="GMT-05:00">
            <uri-template>
                ${trigger_uri_template}/${YEAR}-${MONTH}/${DAY}
            </uri-template>
            <done-flag>${datatype_name_etl}${start_flag_suffix}</done-flag>
        </dataset>
    </datasets>
    
    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${start_date}</instance>
        </data-in>
    </input-events>


    <action>
        <workflow>
            <app-path>${workflow_path}</app-path>
            <configuration>
                <property>
                    <name>year</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'yyyy')}</value>
                </property>
                <property>
                    <name>month</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'MM')}</value>
                </property>
                <property>
                    <name>day</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'dd')}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>
1 REPLY 1

avatar
New Contributor

I think I have found the issue--I have incorrectly specified the date for the dataset "instance" as ${start_date} when it should be ${coord:current(0)}.

I suspect that by replacing:

    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${start_date}</instance>
        </data-in>
    </input-events>

...with...

    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${coord:current(0)}</instance>
        </data-in>
    </input-events>

my issue should be resolved.