Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Why might Oozie use coordinator start date (instead of the coordinator nominal date) for the special variables ${YEAR}, ${MONTH}, and ${DAY}?

Highlighted

Why might Oozie use coordinator start date (instead of the coordinator nominal date) for the special variables ${YEAR}, ${MONTH}, and ${DAY}?

New Contributor

Everything that I have read suggests that ${YEAR}, ${MONTH}, and ${DAY} can be used to specify the uri-template for date-based Oozie coordinator datasets (see, e.g., [1], [2]). However, after starting my coordinator via a bundle (see below) on '2017-08-26', Oozie continues to look in the directory '2017-08/26' for the presence of my trigger file (which has existed since the 26th). A StackOverflow post [3] was able to resolve a similar issue by replacing ${YEAR} with ${coord:formatTime(coord:nominalTime(),'yyyy')}, but this seems to break with the simplicity of using Oozie's EL Time Constants. Am I missing something regarding the proper usage of ${YEAR}, ${MONTH}, and ${DAY} for my coordinator's uri-template?

Thanks!

- Matt

Links:

[1] Oozie - . (2017). Oozie.apache.org. Retrieved 28 August 2017, from https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.html#a5.1._Synchronous_Datasets

[2] Wu, Jui-Chung. (2017). GitHub: YahooArchive/oozie. Retrieved 28 August 2017, from https://github.com/YahooArchive/oozie/wiki/Oozie-Coord-Use-Cases

[3] spijs. (2017). How to set current month, day or year as workflow parameter in a coordinator (on Hue). Stackoverflow.com. Retrieved 28 August 2017, from https://stackoverflow.com/questions/41744794/how-to-set-current-month-day-or-year-as-workflow-parame...

Oozie Version:

Oozie server build version: 4.2.0.2.5.0.0-1245

Bundle Source:

<bundle-app
    xmlns="uri:oozie:bundle:0.2" name="run_all_my_coordinators">
    <controls>
        <kick-off-time>${start_date}</kick-off-time>
    </controls>
    <coordinator name="notify_oozie_start">
        <app-path>${oozieProjectRoot}/notify_oozie_start</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_bundle_datatype}</value>
            </property>
        </configuration>
    </coordinator>


...
    
    <coordinator name="my_coordinator">
        <app-path>${coord_path}</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_datatype_name}</value>
            </property>
            <property>
                <name>datatype_dir</name>
                <value>${my_datatype_dir}</value>
            </property>
        </configuration>
    </coordinator>


...


    <coordinator name="notify_oozie_end">
        <app-path>${oozieProjectRoot}/notify_oozie_end</app-path>
        <configuration>
            <property>
                <name>datatype_name</name>
                <value>${my_bundle_datatype}</value>
            </property>
        </configuration>
    </coordinator>
</bundle-app>

Coordinator Source:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<coordinator-app xmlns="uri:oozie:coordinator:0.5"
    name="my_coordinator"
    start="${start_date}"
    end="${end_date}"
    timezone="GMT-05:00"
    frequency="${coord:days(1)}">
    <controls>
        <concurrency>1</concurrency>
    </controls>
    <datasets>
        <dataset frequency="${coord:days(1)}"
            initial-instance="${initial_instance}"
            name="trigger_dataset_name"
            timezone="GMT-05:00">
            <uri-template>
                ${trigger_uri_template}/${YEAR}-${MONTH}/${DAY}
            </uri-template>
            <done-flag>${datatype_name_etl}${start_flag_suffix}</done-flag>
        </dataset>
    </datasets>
    
    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${start_date}</instance>
        </data-in>
    </input-events>


    <action>
        <workflow>
            <app-path>${workflow_path}</app-path>
            <configuration>
                <property>
                    <name>year</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'yyyy')}</value>
                </property>
                <property>
                    <name>month</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'MM')}</value>
                </property>
                <property>
                    <name>day</name>
                    <value>${coord:formatTime(coord:nominalTime(), 'dd')}</value>
                </property>
            </configuration>
        </workflow>
    </action>
</coordinator-app>
1 REPLY 1
Highlighted

Re: Why might Oozie use coordinator start date (instead of the coordinator nominal date) for the special variables ${YEAR}, ${MONTH}, and ${DAY}?

New Contributor

I think I have found the issue--I have incorrectly specified the date for the dataset "instance" as ${start_date} when it should be ${coord:current(0)}.

I suspect that by replacing:

    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${start_date}</instance>
        </data-in>
    </input-events>

...with...

    <input-events>
        <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name">
            <instance>${coord:current(0)}</instance>
        </data-in>
    </input-events>

my issue should be resolved.

Don't have an account?
Coming from Hortonworks? Activate your account here