we are trying to generate an output from SqlContext and would like to place the results on SFTP server. Please suggest a way. we are using spark 1.4.1 version.
I could save the results on HDFS but we are looking for direct connection from spark to SFTP the results.
Add SPARK-SFTP library as --packages or --jar in your spark-submit command.
I am not sure spark 1.4.1 would be able to handle it. Look at upgrading spark to 1.6.1
Include this package in your Spark Applications using:
> $SPARK_HOME/bin/spark-shell --packages com.springml:spark-sftp_2.10:1.0.1
In your sbt build file, add:
libraryDependencies += "com.springml" % "spark-sftp_2.10" % "1.0.1"
<!-- list of dependencies -->
1bf5b3 | zip | jar ) / Date: 2016-05-27 / License: Apache-2.0 / Scala version: 2.10
7d5b02 | zip | jar ) / Date: 2016-01-11 / License: Apache-2.0 / Scala version: 2.10
View solution in original post
Thank you , it helped. @Amit Nandi