question sftp transfer to hdfs in spark as opposed to using a command in a script in Support Questions

question sftp transfer to hdfs in spark as opposed to using a command in a script in Support Questions https://community.cloudera.com/t5/Support-Questions/sftp-transfer-to-hdfs-in-spark-as-opposed-to-using-a-command/m-p/50896#M48617 I wanted to know how to do sftp transfer to hdfs  in spark 1.6. By loading the data mainly in csv format mid size files a few to maybe 50 gigs size workflow. is this reccomended to do in spark or better from a script. I found  a library <A href="https://github.com/springml/spark-sftp" target="_blank">https://github.com/springml/spark-sftp</A> and wanted to know if this a reccomended way of doing things . One of my problems as well was using this library how would I handle say touch files when I need to read data from a specific data to a specific date. Thanks  I am using spark 1.6 and scala with cloudera manager version around  5.7.2 I think . It is routinely upgraded might be around 5.9 Fri, 16 Sep 2022 11:05:22 GMT gimp077 2022-09-16T11:05:22Z