08-07-2016 07:25 PM
rdd.map(kv => (kv._1, new Set[String]() + kv._2)) .reduceByKey(_ ++ _)
In the above code, what is (kv._1, new Set[String]() + kv._2)) and reduceByKey(_ ++ _).
I know reduceByKey(_+_), but not (_++_).. Please let me know if someone knows this..
08-07-2016 10:02 PM
The first operation makes each value into a set containing that single value. ++ just adds collections together, combining elements of both sets. This is trying to build up a set of all values for each key. It can be written more simply as "groupByKey" really. Even this code could be more compact and efficient.