Support Questions

Find answers, ask questions, and share your expertise

Unable to convert a pyspark dataframe to CSV

avatar
New Contributor

i am using CML jupter notebook. while trying to convert a spark dataframe to CSV getting error like MKDIR cannot create a file

 

Below is the code i have used

spark_df_tx.write('tx.csv')

2 REPLIES 2

avatar
New Contributor

i have also checked in terminal that i am able to create a directory

avatar
Super Collaborator

It's working expected. Please find the below code snippet

>>> columns = ["language","users_count"]
>>> data = [("Java", "20000"), ("Python", "100000"), ("Scala", "3000")]
>>> df = spark.createDataFrame(data).toDF(*columns)
>>> df.write.csv("/tmp/test")
>>> df2=spark.read.csv("/tmp/test/*.csv")
d>>> df2.show()
+------+------+
|   _c0|   _c1|
+------+------+
|Python|100000|
| Scala|  3000|
|  Java| 20000|
+------+------+