Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

In spark, why foreach is designed as an action?

avatar
Explorer

This is a newbie question. I understand that spark actions include count(), first(), take(), etc, each triggers execution with some sort of "output", either to console, or to storage. What I don't understand is why foreach is designed as an action?

3 REPLIES 3

avatar
Super Collaborator

The foreach action in Spark is designed like a forced map (so the "map" action occurs on the executors). Foreach is useful for a couple of operations in Spark. They are required to be used when you want to guarantee an accumulator's value to be correct. In addition, they can be used when you want to move data to an external system, like a database, though typically a foreachPartition is used for that operation.

avatar
Contributor

To add to Joe's response, one more thing to note here is that foreach() action doesn't return anything. So if you want to invoke any transformations on your RDDs on executors without returning anything back to driver, you can invoke them using foreach() action which is not normally possible with other actions apart from saveAsxxx() actions.

-Prav

avatar
New Contributor

You can use RDD.toLocalIterator() to bring the data to the driver (one RDD partition at a time for your low banner ad design cost😞

val array = sc.parallelize(List(1, 2, 3, 4))
for(rec <- array.toLocalIterator) { println(rec) }