question Re: How to run spark df.write inside UDF called in rdd.foreach or rdd.foreachpartition in Support Questions

How to run spark df.write inside UDF called in rdd.foreach or rdd.foreachpartition

Jack_sparrow — Mon, 08 Sep 2025 12:04:24 GMT

How to run spark df.write inside UDF called in rdd.foreach or rdd.foreachpartition

I.e. spark session object inside executor.

Re: How to run spark df.write inside UDF called in rdd.foreach or rdd.foreachpartition

VidyaSargur — Mon, 08 Sep 2025 15:32:10 GMT

@Jack_sparrow, Welcome to our community! To help you get the best possible answer, I have tagged in our Spark experts @haridjh and @vafs, who may be able to assist you further.

Please feel free to provide any additional information or details about your query. We hope that you will find a satisfactory solution to your question.

Re: How to run spark df.write inside UDF called in rdd.foreach or rdd.foreachpartition

vafs — Mon, 08 Sep 2025 20:12:38 GMT

Hello @Jack_sparrow,

Glad to see you on the Community.

As far as I know, df.write is not possible to be used on an rdd.foreach or rdd.foreachpartition.
The reason is because df.write is a driver-side action, it triggers a Spark job.
rdd.foreach or rdd.foreachpartition are executors, and executors cannot trigger jobs.

Check these references:
https://stackoverflow.com/questions/46964250/nullpointerexception-creating-dataset-dataframe-inside-foreachpartition-foreach
https://stackoverflow.com/questions/46964250/nullpointerexception-creating-dataset-dataframe-inside-foreachpartition-foreach
https://sparkbyexamples.com/spark/spark-foreachpartition-vs-foreach-explained

The option that looks like it works for you is this:

df.write.partitionBy

Something like this:

df.write.partitionBy("someColumn").parquet("/path/out")

Re: How to run spark df.write inside UDF called in rdd.foreach or rdd.foreachpartition

Jack_sparrow — Tue, 09 Sep 2025 04:01:05 GMT

Thank you for the response.