question Re: Is there a way to get time-based ticks/triggers in a Spark Streaming job? in Archives of Support Questions (Read Only)

Is there a way to get time-based ticks/triggers in a Spark Streaming job?

rgelhausen — Thu, 21 Apr 2016 00:17:43 GMT

Re: Is there a way to get time-based ticks/triggers in a Spark Streaming job?

bleonhardi — Thu, 21 Apr 2016 05:01:17 GMT

That is one of the things that is more natural in storm. I think your only chance is to set a pretty low base frequency and then either check for the time/trigger event yourself ( in code that gets executed Iike a mappartitions. ) or to use a trigger input ( for example a kafka topic wirh control commands) and join with your main data stream.

the first approach would be in pseudo code

Inputstream.mappartitions{

String command=<load trigger from database hbase whatever...>

Transform your data flow based on command

}

Re: Is there a way to get time-based ticks/triggers in a Spark Streaming job?

rgelhausen — Fri, 05 Aug 2016 14:30:19 GMT

I ended up creating an additional source upstream that generates "tick" events at my specified interval, then joined the two RDDs. Every interval, the RDD element from the "tick" stream has a non-zero value.