02-06-2015 06:49 AM
We were able to successfully Integrate drools with spark, When we try to apply rules from Drools we were able to do for Batch file hich is present in HDFS, But we tried to use drools for Streaming file so that we can make decision instantly, But we couldn't figure out how to do it.Below is the sniipet of the code what we are trying to achieve.
SparkConf conf = new SparkConf().setAppName("sample"); JavaSparkContext sc = new JavaSparkContext(conf); JavaRDD<String> javaRDD = sc.textFile("/user/root/spark/sample.dat"); List<String> store = new ArrayList<String>(); store = javaRDD.collect(); 2nd case when we use streaming context SparkConf sparkconf = new SparkConf().setAppName("sparkstreaming"); JavaStreamingContext ssc = new JavaStreamingContext(sparkconf, new Duration(1)); JavaDStream<String> lines = ssc.socketTextStream("xx.xx.xx.xx", xxxx);
In the first case we were able apply our rules on the variable store, but in the second case we were not able to apply rules on the dstream lines. If someone has some idea, how we can it will be a great help. Thanks in Advance.
02-10-2015 04:04 PM
We have been using drools with CDH but not with spark stream though. If you can share a short test case with pom everything in a zip, we may be able to help you figure out the root cause.