Created 11-28-2017 11:21 PM
I am trying to save a csv to hdfs using quotes, but it does not work. Could you please suggest what i am doing wrong
code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quote","\"").save(output_path)
i am calling it with the following:
--packages com.databricks:spark-csv_2.10:1.5.0 --repositories http://repo.hortonworks.com/content/groups/public/
also tried some other versions, but still writes it without quoting. also tested with other quote character.
#output: aaa,bbb #expected output: "aaa","bbb"
thanks
Created 11-29-2017 11:57 AM
finally i was able to to it:
#code: df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path) #call: --packages com.databricks:spark-csv_2.10:1.5.0
Created 11-29-2017 12:31 AM
Try this option.
.option("quote", ".option("quote", "\u0000")
Created 11-29-2017 11:57 AM
finally i was able to to it:
#code: df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path) #call: --packages com.databricks:spark-csv_2.10:1.5.0