question Re: sftp transfer to hdfs in spark as opposed to using a command in a script in Support Questions

question Re: sftp transfer to hdfs in spark as opposed to using a command in a script in Support Questions https://community.cloudera.com/t5/Support-Questions/sftp-transfer-to-hdfs-in-spark-as-opposed-to-using-a-command/m-p/51034#M48620 <P>I don't know specifically, but yes, it is most likely because the libraries used were not built for distributed system.  For instance, if you had three executors running the code in the library then all three would be reading from the sftp side and directory all vying for the same files and copying them to the destination.  It would be a mess.</P> Thu, 16 Feb 2017 20:38:07 GMT mbigelow 2017-02-16T20:38:07Z