whats the actual difference between the two in terms of performing transformations on the live data coming in and also with data thats already in just few mins ago i mean combining live streaming + sliding window processing?? i.e. combining the data which just arrived few mins ago with the data that is coming in live..
Flink is a pure streaming framework with a lot of windowing capabilities. Spark streaming is still new and just coming from Spark Summit, Databricks is investing in Spark Streaming heavily going forward. Spark Streaming is a micro-batch operation. Flink is not covered by our support and Spark Streaming is, consider that when you make a decision on the framework. At the Summit, Databricks had mentioned that time based aggregations will be a focus for Spark Streaming in the next few releases. Spark Summit website will have slides posted within weeks.
@Artem Ervits thanks for the reply How about storm can we use time based aggregations using storm??? So with existing spark streaming api in 1.5 as its not experimental and 1.6 is experimental can we perform time based aggregations using statefull RDDs, window based transformations in spark streaming??