How to process list of RDDs (List[RDD]) (foreach Alternative)

melvinmendoza — Tue, 14 Feb 2017 15:20:20 GMT

How to process list of RDDs, foreach is sequential is there an alternative?

val param = RDD[(id, testNo, catId, value)] 

val key = param.map(f => (f.catId, f.testNo)).distinct.collect.toList.par

key.foreach( key => { 
// kmeans processing 
// get standard deviation 
// computation 
// etc... etc.. 
})

Re: How to process list of RDDs (List[RDD]) (foreach Alternative)

melvinmendoza — Thu, 28 Sep 2017 15:08:13 GMT

I use Futures and Await. Please see stackoverflow answer.

question How to process list of RDDs (List[RDD]) (foreach Alternative) in Support Questions

How to process list of RDDs (List[RDD]) (foreach Alternative)

Re: How to process list of RDDs (List[RDD]) (foreach Alternative)