Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
Quote data at writing to HDFS does not work
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)
Explorer
Created 11-28-2017 11:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to save a csv to hdfs using quotes, but it does not work. Could you please suggest what i am doing wrong
code:
df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quote","\"").save(output_path)
i am calling it with the following:
--packages com.databricks:spark-csv_2.10:1.5.0 --repositories http://repo.hortonworks.com/content/groups/public/
also tried some other versions, but still writes it without quoting. also tested with other quote character.
#output: aaa,bbb #expected output: "aaa","bbb"
thanks
1 ACCEPTED SOLUTION
Explorer
Created 11-29-2017 11:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
finally i was able to to it:
#code: df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path) #call: --packages com.databricks:spark-csv_2.10:1.5.0
2 REPLIES 2
Rising Star
Created 11-29-2017 12:31 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Try this option.
.option("quote", ".option("quote", "\u0000")
Explorer
Created 11-29-2017 11:57 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
finally i was able to to it:
#code: df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path) #call: --packages com.databricks:spark-csv_2.10:1.5.0
