How to SFTP a generated file in spark 1.4.1

suryavamshivard — Mon, 05 Sep 2016 05:04:26 GMT

we are trying to generate an output from SqlContext and would like to place the results on SFTP server. Please suggest a way. we are using spark 1.4.1 version.

I could save the results on HDFS but we are looking for direct connection from spark to SFTP the results.

Re: How to SFTP a generated file in spark 1.4.1

anandi — Mon, 05 Sep 2016 23:07:22 GMT

Hi Surya,

Add SPARK-SFTP library as --packages or --jar in your spark-submit command.

I am not sure spark 1.4.1 would be able to handle it. Look at upgrading spark to 1.6.1

Check

https://github.com/springml/spark-sftp

https://spark-packages.org/package/springml/spark-sftp

Include this package in your Spark Applications using:

spark-shell, pyspark, or spark-submit

> $SPARK_HOME/bin/spark-shell --packages com.springml:spark-sftp_2.10:1.0.1

sbt

In your sbt build file, add:

libraryDependencies += "com.springml" % "spark-sftp_2.10" % "1.0.1"

Maven

In your pom.xml, add:

<dependencies>
  <!-- list of dependencies -->
  <dependency>
    <groupId>com.springml</groupId>
    <artifactId>spark-sftp_2.10</artifactId>
    <version>1.0.1</version>
  </dependency>
</dependencies>

Releases

1bf5b3 | zip | jar ) / Date: 2016-05-27 / License: Apache-2.0 / Scala version: 2.10

7d5b02 | zip | jar ) / Date: 2016-01-11 / License: Apache-2.0 / Scala version: 2.10

Re: How to SFTP a generated file in spark 1.4.1

suryavamshivard — Tue, 06 Sep 2016 21:13:15 GMT

Thank you , it helped. @Amit Nandi

question How to SFTP a generated file in spark 1.4.1 in Archives of Support Questions (Read Only)

How to SFTP a generated file in spark 1.4.1

Re: How to SFTP a generated file in spark 1.4.1

spark-shell, pyspark, or spark-submit

sbt

Maven

Releases

Re: How to SFTP a generated file in spark 1.4.1