Support Questions

Find answers, ask questions, and share your expertise

How to process list of RDDs (List[RDD]) (foreach Alternative)

avatar
Contributor

How to process list of RDDs, foreach is sequential is there an alternative?

val param = RDD[(id, testNo, catId, value)] 

val key = param.map(f => (f.catId, f.testNo)).distinct.collect.toList.par

key.foreach( key => { 
// kmeans processing 
// get standard deviation 
// computation 
// etc... etc.. 
})
1 ACCEPTED SOLUTION

avatar
Contributor

I use Futures and Await. Please see stackoverflow answer.

View solution in original post

1 REPLY 1

avatar
Contributor

I use Futures and Await. Please see stackoverflow answer.