Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to process list of RDDs (List[RDD]) (foreach Alternative)

Solved Go to solution
Highlighted

How to process list of RDDs (List[RDD]) (foreach Alternative)

Contributor

How to process list of RDDs, foreach is sequential is there an alternative?

val param = RDD[(id, testNo, catId, value)] 

val key = param.map(f => (f.catId, f.testNo)).distinct.collect.toList.par

key.foreach( key => { 
// kmeans processing 
// get standard deviation 
// computation 
// etc... etc.. 
})
1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to process list of RDDs (List[RDD]) (foreach Alternative)

Contributor

I use Futures and Await. Please see stackoverflow answer.

1 REPLY 1

Re: How to process list of RDDs (List[RDD]) (foreach Alternative)

Contributor

I use Futures and Await. Please see stackoverflow answer.

Don't have an account?
Coming from Hortonworks? Activate your account here