Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

In spark, why foreach is designed as an action?

In spark, why foreach is designed as an action?

New Contributor

This is a newbie question. I understand that spark actions include count(), first(), take(), etc, each triggers execution with some sort of "output", either to console, or to storage. What I don't understand is why foreach is designed as an action?

3 REPLIES 3

Re: In spark, why foreach is designed as an action?

Expert Contributor

The foreach action in Spark is designed like a forced map (so the "map" action occurs on the executors). Foreach is useful for a couple of operations in Spark. They are required to be used when you want to guarantee an accumulator's value to be correct. In addition, they can be used when you want to move data to an external system, like a database, though typically a foreachPartition is used for that operation.

Re: In spark, why foreach is designed as an action?

New Contributor

To add to Joe's response, one more thing to note here is that foreach() action doesn't return anything. So if you want to invoke any transformations on your RDDs on executors without returning anything back to driver, you can invoke them using foreach() action which is not normally possible with other actions apart from saveAsxxx() actions.

-Prav

Re: In spark, why foreach is designed as an action?

New Contributor

You can use RDD.toLocalIterator() to bring the data to the driver (one RDD partition at a time for your low banner ad design cost):

val array = sc.parallelize(List(1, 2, 3, 4))
for(rec <- array.toLocalIterator) { println(rec) }
Don't have an account?
Coming from Hortonworks? Activate your account here