question Re: How to save all the output of pyspark sql query into a text file or any file in Support Questions

How to save all the output of pyspark sql query into a text file or any file

barlow — Fri, 16 Sep 2022 13:33:53 GMT

Hello community,

The output from the pyspark query below produces the following output

The pyspark query is as follows:

#%%
import findspark
findspark.init('/home/packt/spark-2.1.0-bin-hadoop2.7')
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName('ops').getOrCreate()
df = spark.read.csv('/home/packt/Downloads/Spark_DataFrames/HumanResources_vEmployeeDepartment.csv',inferSchema=True,header=True)
df.createOrReplaceTempView('HumanResources_vEmployeeDepartment')
myresults = spark.sql("""SELECT
  FirstName
 ,LastName
 ,JobTitle
FROM HumanResources_vEmployeeDepartment
ORDER BY FirstName, LastName DESC""")
myresults.show()

Can someone show me how to save the results to a text / csv file ( or any file please)

Thanks Carlton

Re: How to save all the output of pyspark sql query into a text file or any file

falbani — Mon, 06 Aug 2018 20:21:17 GMT

@Carlton Patterson

You can use this to write whole dataframe to single file:

myresults.coalesce(1).write.csv("/tmp/myresults.csv")

HTH

*** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer.

Re: How to save all the output of pyspark sql query into a text file or any file

barlow — Tue, 07 Aug 2018 03:56:46 GMT

Felix, thank you so much. It worked like a dream

Re: How to save all the output of pyspark sql query into a text file or any file

barlow — Tue, 07 Aug 2018 04:02:55 GMT

Is there a way to get the results with the header info?

Re: How to save all the output of pyspark sql query into a text file or any file

mark_hadoop — Tue, 02 Oct 2018 18:40:46 GMT

@Carlton Patterson

myresults.coalesce(1).write.format('csv').save("/tmp/myresults.csv", header='true')

Re: How to save all the output of pyspark sql query into a text file or any file

shubh — Sat, 13 Jun 2020 05:24:09 GMT

Getting permission denied error :Permission denied: user=cldraproc, access=WRITE, inode="/":yarn:supergroup:drwxr-xr-x

df.coalesce(1).write.format('csv').save("/home/cldraproc/shobhit/ccorp.csv", header='true');

Re: How to save all the output of pyspark sql query into a text file or any file

VidyaSargur — Mon, 15 Jun 2020 07:07:10 GMT

@shubh As this is an older post that has been marked solved in 2018. You would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question.