question Re: Quote data at writing to HDFS does not work in Archives of Support Questions (Read Only)

Quote data at writing to HDFS does not work

ztalas1979 — Wed, 29 Nov 2017 07:21:45 GMT

I am trying to save a csv to hdfs using quotes, but it does not work. Could you please suggest what i am doing wrong

code:

df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quote","\"").save(output_path)

i am calling it with the following:

--packages com.databricks:spark-csv_2.10:1.5.0 --repositories http://repo.hortonworks.com/content/groups/public/

also tried some other versions, but still writes it without quoting. also tested with other quote character.

#output:
aaa,bbb

#expected output:
"aaa","bbb"

thanks

Re: Quote data at writing to HDFS does not work

MindGlass — Wed, 29 Nov 2017 08:31:07 GMT

Try this option.

.option("quote", ".option("quote", "\u0000")

Re: Quote data at writing to HDFS does not work

ztalas1979 — Wed, 29 Nov 2017 19:57:16 GMT

finally i was able to to it:

#code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path)

#call:
--packages com.databricks:spark-csv_2.10:1.5.0