- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
saveAsTextFile creates deflate file on the cluster, I would like to create pure text file ,how would I do that?
- Labels:
-
Apache Spark
Created ‎09-06-2016 02:10 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
when I do the saveAsTextFile on the local machine , text files are created but when I do the same on cluster it creates .deflate files, how can I have text files on the cluster , please help.
sqlContext.sql(sourceQuery).map { row => FixedLengthParser.parseRecord(row) }.repartition(1).saveAsTextFile(s"$outputLocalPath/$sourceId")
Created ‎09-06-2016 02:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Set hive.exec.compress.output to false. Your cluster may be configured for compression by default, with the default codec.
Alternatively, keep the compressed output but view it with the "hdfs dfs -text <file path>" command.
Created ‎09-06-2016 03:27 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
Thanks for the reply. I tried setting the value at run time, but no luck. also I am not using hiveContext but using only sqlContext, do I need to set any other parameter also?
spark-submit --class com.rbc.aml.dataprocessing.CreateGff --master local[1] /users/qjgvws3/spark/dataprocessing-0.0.1-SNAPSHOT-jar-with-dependencies.jar --conf hive.exec.compress.output=false |
