Hi experts, I have some basic questions about the relationship between MapReduce and HDFS: The Data File placing on HDFS is through MapReduce? All transactions in HDFS are using MapReduce jobs? Anyone knows the answer? Many thanks! ... View more

Hi experts, I have attached to this post dataset sample.txt And I am trying to extract some association rules using Spark Mllib: val transactions = sc.textFile("DATA") import org.apache.spark.mllib.fpm.AssociationRules import org.apache.spark.mllib.fpm.FPGrowth.FreqItemset val freqItemsets = transactions.map(_.split(",")).flatMap(xs => (xs.combinations(1) ++ xs.combinations(2) ++ xs.combinations(3) ++ xs.combinations(4) ++ xs.combinations(5)).filter(_.nonEmpty).map(x => (x.toList, 1L)) ).reduceByKey(_ + _).map{case (xs, cnt) => new FreqItemset(xs.toArray, cnt)} val ar = new AssociationRules().setMinConfidence(0.8) val results = ar.run(freqItemsets)results.collect().foreach { rule => println("[" + rule.antecedent.mkString(",") + "=>" + rule.consequent.mkString(",") + "]," + rule.confidence)} However, my code returns a dozen rules with confidence equal to 1 ... which makes little sense! Does anyone know if I lack some parameterization? ... View more

Hi experts, I've this line from a .txt which results from a Group Operator: 1;(7287026502032012,18);{(706)};{(101200010)};{(17286)};{(oz)};2.5 Basically I've 7 fields how can I obtain this: 1;7287026502032012,18;706;101200010;17286;oz;2.5 Many thanks! ... View more

Having this statement: Values = FILTER Input_Data BY Fields > 0 How to cont the number of records that was filtered and not? Many thanks! ... View more

Hi, Everytime that I run my Pig Script it generates a multiple files in HDFS (I never know the number). I need to do some anlytics using Spark. How can I join that multiple files to have only one file like: val data = sc.textFile("PATH/Filejoined"); Thanks! ... View more