https://community.cloudera.com/t5/Support-Questions/Unable-to-Create-a-single-file-with-PySpark-query/m-p/213877#M175801 <P><A rel="user" href="https://community.cloudera.com/users/63166/carlton.html" nodeid="63166">@Carlton Patterson</A> </P><P>This is <STRONG>not possible with default save/csv/json</STRONG> functions but using <A href="https://stackoverflow.com/questions/40792434/spark-dataframe-save-in-single-file-on-hdfs-location" target="_blank">Hadoop API</A><STRONG></STRONG> we can rename the filename.</P><P><STRONG><U>Example:</U></STRONG></P><PRE>&gt;&gt;&gt; df=spark.sql("select int(1)id,string('ll')name") //create a dataframe &gt;&gt;&gt; df.coalesce(1).write.mode("overwrite").csv("/user/shu/test/temp_dir") //writing the df to temp-dir &gt;&gt;&gt; from py4j.java_gateway import java_import &gt;&gt;&gt; java_import(spark._jvm, 'org.apache.hadoop.fs.Path') &gt;&gt;&gt; fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration()) &gt;&gt;&gt; file = fs.globStatus(sc._jvm.Path('/user/shu/test/temp_dir/part*'))[0].getPath().getName() //get the filename of temp_dir &gt;&gt;&gt; fs.rename(sc._jvm.Path('/user/shu/test/temp_dir/' + file),sc._jvm.Path('/user/shu/test/mydata.csv')) //rename the temp directory file with desired filename and directory path &gt;&gt;&gt; fs.delete(sc._jvm.Path('/user/shu/test/temp_dir'), True) //delete the temp directory.</PRE><P>-</P><P>If the Answer helped to resolve your issue, <STRONG>Click on Accept button below to accept the answer,</STRONG> That would be great help to Community users to find solution quickly for these kind of issues.</P> Sat, 20 Oct 2018 23:34:33 GMT Shu_ashu 2018-10-20T23:34:33Z