question Patterns for batch processing time-series data? in Archives of Support Questions (Read Only)

Patterns for batch processing time-series data?

pcoates — Thu, 19 Nov 2015 21:16:49 GMT

What patterns or practices exist for dealing with time-series data specifically in batch mode, i.e, Tez or MR as opposed to Spark. Sorting orders the data within a block or ORC split, but how are boundaries between blocks usually handled? For instance, finding derivatives, inflection points, etc. breaks down at file boundaries---are there standard patterns or libraries to deal with this?

Re: Patterns for batch processing time-series data?

nsabharwal — Fri, 20 Nov 2015 02:17:26 GMT

@Brandon Wilson

Re: Patterns for batch processing time-series data?

bwilson — Sat, 21 Nov 2015 01:25:13 GMT

One option that comes to mind is to leverage a custom InputFormat. HDFS doesn't care about where it breaks a file so the input format helps ensure that there are not awkward breaks between blocks when reading files. With this approach, you can define your own notion of a record, whether it be a line of text (TextInputFormat) or a window that could encapsulate multiple records.

You can then use this custom InputFormat to read the data into An MR job or you can use it to develop you own custom Pig loader to work with your data in Pig.

I am not personally aware of any libraries that have been built to address time-series specifically.

Re: Patterns for batch processing time-series data?

nsabharwal — Sat, 21 Nov 2015 05:09:12 GMT

@Brandon Wilson Thanks! 🙂