Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Quote data at writing to HDFS does not work

Solved Go to solution

Quote data at writing to HDFS does not work

New Contributor

I am trying to save a csv to hdfs using quotes, but it does not work. Could you please suggest what i am doing wrong

code:

df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quote","\"").save(output_path)

i am calling it with the following:

--packages com.databricks:spark-csv_2.10:1.5.0 --repositories http://repo.hortonworks.com/content/groups/public/

also tried some other versions, but still writes it without quoting. also tested with other quote character.

#output:
aaa,bbb

#expected output:
"aaa","bbb"

thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Quote data at writing to HDFS does not work

New Contributor

finally i was able to to it:

#code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path)

#call:
--packages com.databricks:spark-csv_2.10:1.5.0
2 REPLIES 2
Highlighted

Re: Quote data at writing to HDFS does not work

New Contributor

Try this option.

.option("quote", ".option("quote", "\u0000")

Re: Quote data at writing to HDFS does not work

New Contributor

finally i was able to to it:

#code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path)

#call:
--packages com.databricks:spark-csv_2.10:1.5.0
Don't have an account?
Coming from Hortonworks? Activate your account here