Created 08-28-2017 02:57 PM
Everything that I have read suggests that ${YEAR}, ${MONTH}, and ${DAY} can be used to specify the uri-template for date-based Oozie coordinator datasets (see, e.g., [1], [2]). However, after starting my coordinator via a bundle (see below) on '2017-08-26', Oozie continues to look in the directory '2017-08/26' for the presence of my trigger file (which has existed since the 26th). A StackOverflow post [3] was able to resolve a similar issue by replacing ${YEAR} with ${coord:formatTime(coord:nominalTime(),'yyyy')}, but this seems to break with the simplicity of using Oozie's EL Time Constants. Am I missing something regarding the proper usage of ${YEAR}, ${MONTH}, and ${DAY} for my coordinator's uri-template?
Thanks!
- Matt
Links:
[1] Oozie - . (2017). Oozie.apache.org. Retrieved 28 August 2017, from https://oozie.apache.org/docs/4.3.0/CoordinatorFunctionalSpec.html#a5.1._Synchronous_Datasets
[2] Wu, Jui-Chung. (2017). GitHub: YahooArchive/oozie. Retrieved 28 August 2017, from https://github.com/YahooArchive/oozie/wiki/Oozie-Coord-Use-Cases
[3] spijs. (2017). How to set current month, day or year as workflow parameter in a coordinator (on Hue). Stackoverflow.com. Retrieved 28 August 2017, from https://stackoverflow.com/questions/41744794/how-to-set-current-month-day-or-year-as-workflow-parame...
Oozie Version:
Oozie server build version: 4.2.0.2.5.0.0-1245
Bundle Source:
<bundle-app xmlns="uri:oozie:bundle:0.2" name="run_all_my_coordinators"> <controls> <kick-off-time>${start_date}</kick-off-time> </controls> <coordinator name="notify_oozie_start"> <app-path>${oozieProjectRoot}/notify_oozie_start</app-path> <configuration> <property> <name>datatype_name</name> <value>${my_bundle_datatype}</value> </property> </configuration> </coordinator> ... <coordinator name="my_coordinator"> <app-path>${coord_path}</app-path> <configuration> <property> <name>datatype_name</name> <value>${my_datatype_name}</value> </property> <property> <name>datatype_dir</name> <value>${my_datatype_dir}</value> </property> </configuration> </coordinator> ... <coordinator name="notify_oozie_end"> <app-path>${oozieProjectRoot}/notify_oozie_end</app-path> <configuration> <property> <name>datatype_name</name> <value>${my_bundle_datatype}</value> </property> </configuration> </coordinator> </bundle-app>
Coordinator Source:
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <coordinator-app xmlns="uri:oozie:coordinator:0.5" name="my_coordinator" start="${start_date}" end="${end_date}" timezone="GMT-05:00" frequency="${coord:days(1)}"> <controls> <concurrency>1</concurrency> </controls> <datasets> <dataset frequency="${coord:days(1)}" initial-instance="${initial_instance}" name="trigger_dataset_name" timezone="GMT-05:00"> <uri-template> ${trigger_uri_template}/${YEAR}-${MONTH}/${DAY} </uri-template> <done-flag>${datatype_name_etl}${start_flag_suffix}</done-flag> </dataset> </datasets> <input-events> <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name"> <instance>${start_date}</instance> </data-in> </input-events> <action> <workflow> <app-path>${workflow_path}</app-path> <configuration> <property> <name>year</name> <value>${coord:formatTime(coord:nominalTime(), 'yyyy')}</value> </property> <property> <name>month</name> <value>${coord:formatTime(coord:nominalTime(), 'MM')}</value> </property> <property> <name>day</name> <value>${coord:formatTime(coord:nominalTime(), 'dd')}</value> </property> </configuration> </workflow> </action> </coordinator-app>
Created 08-28-2017 03:14 PM
I think I have found the issue--I have incorrectly specified the date for the dataset "instance" as ${start_date} when it should be ${coord:current(0)}.
I suspect that by replacing:
<input-events> <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name"> <instance>${start_date}</instance> </data-in> </input-events>
...with...
<input-events> <data-in name="datatype_input_event_trigger" dataset="trigger_dataset_name"> <instance>${coord:current(0)}</instance> </data-in> </input-events>
my issue should be resolved.