Support Questions
Find answers, ask questions, and share your expertise

Quote data at writing to HDFS does not work

Explorer

I am trying to save a csv to hdfs using quotes, but it does not work. Could you please suggest what i am doing wrong

code:

df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quote","\"").save(output_path)

i am calling it with the following:

--packages com.databricks:spark-csv_2.10:1.5.0 --repositories http://repo.hortonworks.com/content/groups/public/

also tried some other versions, but still writes it without quoting. also tested with other quote character.

#output:
aaa,bbb

#expected output:
"aaa","bbb"

thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Quote data at writing to HDFS does not work

Explorer

finally i was able to to it:

#code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path)

#call:
--packages com.databricks:spark-csv_2.10:1.5.0

View solution in original post

2 REPLIES 2

Re: Quote data at writing to HDFS does not work

Explorer

Try this option.

.option("quote", ".option("quote", "\u0000")

Re: Quote data at writing to HDFS does not work

Explorer

finally i was able to to it:

#code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path)

#call:
--packages com.databricks:spark-csv_2.10:1.5.0

View solution in original post