question Re: Coping files from Remote server to HDFS in Support Questions

Coping files from Remote server to HDFS

Vinitkumar — Mon, 09 Jul 2018 16:26:34 GMT

Hi,

I have a remote server and Kerberos authenticated Hadoop environment.

I want to copy files from Remote server to HDFS for processing using Spark. Please advise efficient approach/HDFS command to copy files from remote server to HDFS. Any example will be helpful. We are bound by not to use flume or Nifi.

Please note Kerberos is installed on Remote server.

Re: Coping files from Remote server to HDFS

elemesilva — Wed, 11 Jul 2018 00:31:10 GMT

@Vinit Pandey

I suggest on your HDFS server you start a process (using shellscript for example) to execute kinit and after that, get these remote files using sftp or scp example

# scp user@remoteserver:/remotepath/files localpath/

and

# hdfs dfs -put localpath/files /hdfspath

Note: To automate this process you can create a private/public ssh between these servers and create a crontab entry.