Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Quote data at writing to HDFS does not work

avatar
Explorer

I am trying to save a csv to hdfs using quotes, but it does not work. Could you please suggest what i am doing wrong

code:

df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quote","\"").save(output_path)

i am calling it with the following:

--packages com.databricks:spark-csv_2.10:1.5.0 --repositories http://repo.hortonworks.com/content/groups/public/

also tried some other versions, but still writes it without quoting. also tested with other quote character.

#output:
aaa,bbb

#expected output:
"aaa","bbb"

thanks

1 ACCEPTED SOLUTION

avatar
Explorer

finally i was able to to it:

#code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path)

#call:
--packages com.databricks:spark-csv_2.10:1.5.0

View solution in original post

2 REPLIES 2

avatar
Rising Star

Try this option.

.option("quote", ".option("quote", "\u0000")

avatar
Explorer

finally i was able to to it:

#code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path)

#call:
--packages com.databricks:spark-csv_2.10:1.5.0