Support Questions

Find answers, ask questions, and share your expertise

saveAsTextFile creates deflate file on the cluster, I would like to create pure text file ,how would I do that?

when I do the saveAsTextFile on the local machine , text files are created but when I do the same on cluster it creates .deflate files, how can I have text files on the cluster , please help.

sqlContext.sql(sourceQuery).map { row => FixedLengthParser.parseRecord(row) }.repartition(1).saveAsTextFile(s"$outputLocalPath/$sourceId")


Expert Contributor

Set hive.exec.compress.output to false. Your cluster may be configured for compression by default, with the default codec.

Alternatively, keep the compressed output but view it with the "hdfs dfs -text <file path>" command.


Thanks for the reply. I tried setting the value at run time, but no luck. also I am not using hiveContext but using only sqlContext, do I need to set any other parameter also?

spark-submit --class com.rbc.aml.dataprocessing.CreateGff --master local[1] /users/qjgvws3/spark/dataprocessing-0.0.1-SNAPSHOT-jar-with-dependencies.jar --conf hive.exec.compress.output=false