@vamsi valiveti Oozie can materialize coordinator actions (i.e. start tasks/jobs) based on time-based intervals or triggers. For example run Job X every day at 12pm. However, time is not always the only dependency. Sometimes we may want to start a job after all the necessary data is available. So, Oozie coordintor allows us to use both, time and data dependencies, to start a workflow. “dataset”, “input-events” and “output-events” are the pillars for configuring data dependencies in coordinator.xml. Dataset: A “dataset” is essentially an entity that represents data produced by an application and is often defined using it’s directory location. When data is ready to be consumed a file named “_SUCCESS” is added to the folder by default. Alternatively, we can specify the name of the file we want to write instead of “_SUCCESS” by setting the “done-flag”. If we look at your config example you have above, we expect a new dataset to be generated every 60 minutes and the folder will be “tmp/revenue_feed/${YEAR}/${MONTH}/${DAY}/${HOUR}” <dataset name="input1" frequency="60" initial-instance="2009-01-01T00:00Z" timezone="UTC">
<uri-template>
hdfs://localhost:9000/tmp/revenue_feed/${YEAR}/${MONTH}/${DAY}/${HOUR}
</uri-template>
</dataset>
Input-Event: An “input-event” describes the necessary instances of data to start a job. For example, when a coordinator runs, the input-event can be used to check if a “_SUCCESS” has been posted in the last hour, and process the data for that. If nothing matches this criteria, or all the data is more than an hour old, then the job is not executed. Another example is to wait until several files/data instances have been completed before running. We specify the time window to look for these data instances by using the “start-instance” and the “end-instance” So, in the example you have above, it specifies we should process files for the last 24 hours (“current(0)” being the current hour and “current(-23)” being 23 hours ago) <input-events>
<data-in name="coordInput1" dataset="input1">
<start-instance>${coord:current(-23)}</start-instance>
<end-instance>${coord:current(0)}</end-instance>
</data-in>
</input-events>
Output-Event: An “output-event” is the opposite of an “input-event”. It is the output file the job write once it completes. This file can be used as the “input-event” for another coordinator.
... View more
