Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why might Oozie use coordinator start date (instead of the coordinator nominal date) for the special variables ${YEAR}, ${MONTH}, and ${DAY}?

avatar
New Contributor

Everything that I have read suggests that ${YEAR}, ${MONTH}, and ${DAY} can be used to specify the uri-template for date-based Oozie coordinator datasets (see, e.g., [1], [2]). However, after starting my coordinator via a bundle (see below) on '2017-08-26', Oozie continues to look in the directory '2017-08/26' for the presence of my trigger file (which has existed since the 26th). A StackOverflow post [3] was able to resolve a similar issue by replacing ${YEAR} with ${coord:formatTime(coord:nominalTime(),'yyyy')}, but this seems to break with the simplicity of using Oozie's EL Time Constants. Am I missing something regarding the proper usage of ${YEAR}, ${MONTH}, and ${DAY} for my coordinator's uri-template?

Thanks!

- Matt

Links:

[1] Oozie - . (2017). Oozie.apache.org. Retrieved 28 August 2017, from https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.html#a5.1._Synchronous_Datasets

[2] Wu, Jui-Chung. (2017). GitHub: YahooArchive/oozie. Retrieved 28 August 2017, from https://github.com/YahooArchive/oozie/wiki/Oozie-Coord-Use-Cases

[3] spijs. (2017). How to set current month, day or year as workflow parameter in a coordinator (on Hue). Stackoverflow.com. Retrieved 28 August 2017, from https://stackoverflow.com/questions/41744794/how-to-set-current-month-day-or-year-as-workflow-parame...

Oozie Version:

Oozie server build version: 4.2.0.2.5.0.0-1245

Bundle Source:

<bundle-app
    xmlns="uri:oozie:bundle:0.2" name="run_all_my_coordinators">
    <controls>
        <kick-off-time>${start_date}</kick-off-time>
    </controls>
    <coordinator name="notify_oozie_start">
        <app-path>${oozieProjectRoot}/notify_oozie_start</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_bundle_datatype}</value>
            </property>
        </configuration>
    </coordinator>


...
    
    <coordinator name="my_coordinator">
        <app-path>${coord_path}</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_datatype_name}</value>
            </property>
            <property>
                <name>datatype_dir</name>
                <value>${my_datatype_dir}</value>
            </property>
        </configuration>
    </coordinator>


...


    <coordinator name="notify_oozie_end">
        <app-path>${oozieProjectRoot}/notify_oozie_end</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_bundle_datatype}</value>
            </property>
        </configuration>
    </coordinator>
</bundle-app>

Coordinator Source:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<coordinator-app xmlns="uri:oozie:coordinator:0.5"
    name="my_coordinator"
    start="${start_date}"
    end="${end_date}"
    timezone="GMT-05:00"
    frequency="${coord:days(1)}">
    <controls>
        <concurrency>1</concurrency>
    </controls>
    <datasets>
        <dataset frequency="${coord:days(1)}"
            initial-instance="${initial_instance}"
            name="trigger_dataset_name"
            timezone="GMT-05:00">
            <uri-template>
                ${trigger_uri_template}/${YEAR}-${MONTH}/${DAY}
            </uri-template>
            <done-flag>${datatype_name_etl}${start_flag_suffix}</done-flag>
        </dataset>
    </datasets>
    
    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${start_date}</instance>
        </data-in>
    </input-events>


    <action>
        <workflow>
            <app-path>${workflow_path}</app-path>
            <configuration>
                <property>
                    <name>year</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'yyyy')}</value>
                </property>
                <property>
                    <name>month</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'MM')}</value>
                </property>
                <property>
                    <name>day</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'dd')}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>
1 REPLY 1

avatar
New Contributor

I think I have found the issue--I have incorrectly specified the date for the dataset "instance" as ${start_date} when it should be ${coord:current(0)}.

I suspect that by replacing:

    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${start_date}</instance>
        </data-in>
    </input-events>

...with...

    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${coord:current(0)}</instance>
        </data-in>
    </input-events>

my issue should be resolved.