https://community.cloudera.com/t5/Archives-of-Support-Questions/Apache-SPARK-Overwrite-data-file/m-p/105254#M42339 <P>RDD's saveAsTextFile does not give us the opportunity to do that (DataFrame's have "save modes" for things like append/overwrite/ignore). You'll have to control this prior before (maybe delete or rename existing data) or afterwards (write the RDD as a diff dir and then swap it out).</P> Thu, 06 Oct 2016 23:46:07 GMT LesterMartin 2016-10-06T23:46:07Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Apache-SPARK-Overwrite-data-file/m-p/105253#M42338 <P>Hi experts, How can I overwrite an existing file by a new one (data update). Imagine that I've this: </P><OL><LI>result.map(pair =&gt; pair.swap).sortByKey(true).saveAsTextFile("FILE/results") </LI></OL><P>And Imagine that I want to do this:</P><OL><LI>test.map(pair =&gt; pair.swap).sortByKey(false).saveAsTextFile("FILE/results")</LI></OL><P> How can I overwrite the results of the var result to the results of the val test in same directory?</P> Fri, 30 Sep 2016 01:38:59 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Apache-SPARK-Overwrite-data-file/m-p/105253#M42338 prodgers125 2016-09-30T01:38:59Z https://community.cloudera.com/t5/Archives-of-Support-Questions/Apache-SPARK-Overwrite-data-file/m-p/105254#M42339 <P>RDD's saveAsTextFile does not give us the opportunity to do that (DataFrame's have "save modes" for things like append/overwrite/ignore). You'll have to control this prior before (maybe delete or rename existing data) or afterwards (write the RDD as a diff dir and then swap it out).</P> Thu, 06 Oct 2016 23:46:07 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/Apache-SPARK-Overwrite-data-file/m-p/105254#M42339 LesterMartin 2016-10-06T23:46:07Z