@Benjamin Leonhardi Thank you and how about using Spark Streaming & Storm both for this usecase? Not sure if we can integrate spark streaming with Storm may be weird architecture. I think storm as well fits this requirement but we are planning to create a POC on both Storm & spark streaming and then productionize one. Thank you ... View more

@Benjamin Leonhardi Thanks for the answer. Do you know if storm 1.0 can integrate with Machine learning algorithms. Also i believe storm is event driven as well??? Actually iam looking for a technology to use either Spark or storm which solves event driven and low latency problem like if a customer is on the website then our streaming application based on some machine learning models present offers like next best product or next best offer in milli second latency to the customer before he moves away from the website. This is the high level requirement. Do you think we can use both spark streaming and storm as we cannot get milli second latency & event processing with spark streaming. Thank you ... View more

@Dave Russell thanks for your reply dave. In terms of latency i believe its 1sec but i ll know more about the exact latency this week. How about event driven can spark streaming perform event driven real time analytics as well? thanks ... View more

thank you @Neeraj Sabharwal ... View more

Thank you @Neeraj Sabharwal. I ll give this a try. Why do we need EvaluateXPath here? Also i beleive at the end use puthdfs to route the data into hdfs. I beleive currently nifi doesnt have a processor to ingest directly into hive table i can see it has putsql. So i beleive best approach is to use puthdfs. Please let me know thanks.. ... View more

thanks for your reply @Artem Ervits will implement the poc thanks ... View more

thanks @Ancil McBarnett. will have a look..:) ... View more

@Artem Ervits just wondering once the zip files are in hdfs and if they are streaming into hdfs using nifi as zip format i believe we need a way to automate the unzip process and analyse the files stored in zip folder.. Any ideas how we can acheive this please?? thank you ... View more