https://community.cloudera.com/t5/Support-Questions/How-to-save-each-partition-of-a-Dataframe-Dataset-in/m-p/46811#M24813 <P>Hello</P><P>I currently use spark 2.0.1 and i try to save my dataset into a "partitioned table Hive" with insertInto() or on S3 storage with partitionBy("col") with job in concurrency (parallel). But with this 2 methods each partition of my dataset is save sequentially one by one . It's very very SLOW.</P><P>I already know that I must use insertInto() or partitionBy() one at time.</P><P>I assume that in spark.2.0.1 Dataframe are Resilient Data Set .</P><P>My current code :</P><PRE>df.write.mode(SaveMode.Append).partitionBy("col").save("s3://bucket/diroutput")</PRE><P>Or</P><PRE>df.write.mode(SaveMode.Append).insertInto("TableHivealreadypartitioned")</PRE><P>So I try some stuff with df.foreachPartition like this :</P><PRE>df.foreachPartition{datasetpartition =&gt; datasetpartition.foreach(row =&gt; row.sometransformation)}</PRE><P>Unfortunately i still do not find a way to write/save in parallel each partition of my dataset.</P><P>Someone already done this?</P><P>Can you tell me how to proceed?</P><P>Is it a wrong direction?</P><P>thanks for your help</P> Fri, 16 Sep 2022 10:46:14 GMT damdr 2022-09-16T10:46:14Z