question datasets and output input events : what correlation between YEAR/MONTH/DAY and the instance ? in Archives of Support Questions (Read Only)

datasets and output input events : what correlation between YEAR/MONTH/DAY and the instance ?

sebastien_frack — Wed, 04 Oct 2017 23:39:07 GMT

Maybe it is obvious but I was wondering :

When we declare a dataset, based on the date ($YEAR/$MONTH/$DAY/data for example) as an output-events, and used from an input-events where "instance" will watch at current(0) :

Does the dated directory name is directly used to check the input event, or is there a kind of database that register that inside Oozie ? In other words, if we don't mention the output-events and create the "good" directory, will it still working ?

Re: datasets and output input events : what correlation between YEAR/MONTH/DAY and the instance ?

kgautam — Sat, 07 Oct 2017 22:59:41 GMT

Yess it will

1. Generally at ingestion stage data is collected at minute, hourly or daily level.

2. To keep data together based on timestamp, one follow "hdfs path" naming convention as /a/b/b/yyyy/mm/dd

3. the job which consumes this data for performing ETL , needs to choose a range of this path like a week , or a month etc hence datasets have YYYY/MM/DD as the variable param in them .

Re: datasets and output input events : what correlation between YEAR/MONTH/DAY and the instance ?

sebastien_frack — Mon, 09 Oct 2017 14:24:52 GMT

thanks 🙂