finally i was able to to it: #code: df.write.format('com.databricks.spark.csv').mode('overwrite').option("header", "false").option("quoteMode", "ALL").save(output_path) #call: --packages com.databricks:spark-csv_2.10:1.5.0 ... View more

found the issue. It is exactly the same problem mentioned in the below link: https://github.com/tweag/sparkle/issues/105 went back to spark 1.6 and it works ... View more