About suryavamshivard

suryavamshivard · ‎09-06-2016

Hi, Thanks for the reply. I tried setting the value at run time, but no luck. also I am not using hiveContext but using only sqlContext, do I need to set any other parameter also? spark-submit --class com.rbc.aml.dataprocessing.CreateGff --master local[1] /users/qjgvws3/spark/dataprocessing-0.0.1-SNAPSHOT-jar-with-dependencies.jar --conf hive.exec.compress.output=false

suryavamshivard · ‎09-06-2016

Thank you , it helped. @Amit Nandi

suryavamshivard · ‎09-06-2016

when I do the saveAsTextFile on the local machine , text files are created but when I do the same on cluster it creates .deflate files, how can I have text files on the cluster , please help. sqlContext.sql(sourceQuery).map { row => FixedLengthParser.parseRecord(row) }.repartition(1).saveAsTextFile(s"$outputLocalPath/$sourceId")

suryavamshivard · ‎09-04-2016

we are trying to generate an output from SqlContext and would like to place the results on SFTP server. Please suggest a way. we are using spark 1.4.1 version. I could save the results on HDFS but we are looking for direct connection from spark to SFTP the results.

Online	Offline
Last Visited	‎05-18-2017 07:37 PM

Member Since	‎05-10-2016 02:31 PM
Last Visited	‎05-18-2017 07:37 PM
Posts	7
Kudos received	1

Cloudera Community

Re: saveAsTextFile creates deflate file on the clu...

Re: How to SFTP a generated file in spark 1.4.1

saveAsTextFile creates deflate file on the cluster...

How to SFTP a generated file in spark 1.4.1