Support Questions

Find answers, ask questions, and share your expertise

saveAsTextFile creates deflate file on the cluster, I would like to create pure text file ,how would I do that?

avatar

when I do the saveAsTextFile on the local machine , text files are created but when I do the same on cluster it creates .deflate files, how can I have text files on the cluster , please help.

sqlContext.sql(sourceQuery).map { row => FixedLengthParser.parseRecord(row) }.repartition(1).saveAsTextFile(s"$outputLocalPath/$sourceId")

2 REPLIES 2

avatar
Super Collaborator

Set hive.exec.compress.output to false. Your cluster may be configured for compression by default, with the default codec.

Alternatively, keep the compressed output but view it with the "hdfs dfs -text <file path>" command.

avatar

Hi,

Thanks for the reply. I tried setting the value at run time, but no luck. also I am not using hiveContext but using only sqlContext, do I need to set any other parameter also?

spark-submit --class com.rbc.aml.dataprocessing.CreateGff --master local[1] /users/qjgvws3/spark/dataprocessing-0.0.1-SNAPSHOT-jar-with-dependencies.jar --conf hive.exec.compress.output=false