Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎08-07-2016
Accepted Solution

reduceByKey(_ ++ _)

rdd.map(kv => (kv._1, new Set[String]() + kv._2)) .reduceByKey(_ ++ _)

 

In the above code, what is (kv._1, new Set[String]() + kv._2)) and reduceByKey(_ ++ _).

 

I know reduceByKey(_+_), but not (_++_).. Please let me know if someone knows this..

 

Thanks!

Cloudera Employee
Posts: 481
Registered: ‎08-11-2014

Re: reduceByKey(_ ++ _)

The first operation makes each value into a set containing that single value. ++ just adds collections together, combining elements of both sets. This is trying to build up a set of all values for each key. It can be written more simply as "groupByKey" really. Even this code could be more compact and efficient.