question reduceByKey(_ ++ _) in Archives of Support Questions (Read Only)

reduceByKey(_ ++ _)

jee1 — Mon, 08 Aug 2016 02:25:32 GMT

rdd.map(kv => (kv._1, new Set[String]() + kv._2)) .reduceByKey(_ ++ _)

In the above code, what is (kv._1, new Set[String]() + kv._2)) and reduceByKey(_ ++ _).

I know reduceByKey(_+_), but not (_++_).. Please let me know if someone knows this..

Thanks!

Re: reduceByKey(_ ++ _)

srowen — Mon, 08 Aug 2016 05:02:33 GMT

The first operation makes each value into a set containing that single value. ++ just adds collections together, combining elements of both sets. This is trying to build up a set of all values for each key. It can be written more simply as "groupByKey" really. Even this code could be more compact and efficient.