Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark : Equivalent of RDD action foreach

Highlighted

Spark : Equivalent of RDD action foreach

Explorer

We are new to spark looking for an equivalent action of an RDD similar to foreach which applies to each element

2 REPLIES 2

Re: Spark : Equivalent of RDD action foreach

Expert Contributor

Theres not really an rdd equivalent like "foreachRDD" in core spark but there is a foreach partition that allows you to apply function foreach partition in the RDD.

In spark streaming there is a foreachRDD which allows you to apply a function to each "RDD" in the DStream.

//Core Spark
foreachPartition(f: (Iterator[T]) ⇒ Unit): Unit
//Spark-Streaming
foreachRDD(foreachFunc: (RDD[T]) ⇒ Unit): Unit
Highlighted

Re: Spark : Equivalent of RDD action foreach

New Contributor

Interesting, assumed that something like the following:

sc.parallelize([1, 2, 3, 4]).foreach(lambda x: accum.add(x))

As per the Spark manual would execute in parallel in Spark, applying a function to each element. But perhaps that is not quite what the above line is doing.

Don't have an account?
Coming from Hortonworks? Activate your account here