question Re: how to copy file from remote server to HDFS in Support Questions

how to copy file from remote server to HDFS

mike_bronson7 — Tue, 07 Jul 2020 16:57:17 GMT

I have a remote server and servers authenticated Hadoop environment.

I want to copy file from Remote server to Hadoop machine to HDFS

Please advise efficient approach/HDFS command to copy files from remote server to HDFS.

Any example will be helpful.

as ordinary way to copy file from remote server to server itself is

scp -rp file remote_server:/tmp

but this approach not support copy directly to hdfs

Re: how to copy file from remote server to HDFS

vsundaram — Tue, 14 Jul 2020 19:40:58 GMT

@mike_bronson7 Are you trying to copy local files from your remote machine to destination HDFS cluster? You could use distcp if it is between hdfs. Please refer below documentation.

https://docs.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_data_cluster_migrate.html

For the local files in remote machine, you could SCP the files to any of the cluster node which has hdfs client installed and the do a "-copyFromLocal" or "-put" to push that to HDFS. Hope this helps.

Re: how to copy file from remote server to HDFS

SujaP — Fri, 17 Dec 2021 02:03:28 GMT

Can you please give an example for this?

Thanks

Re: how to copy file from remote server to HDFS

BrunoAndrade — Wed, 17 Apr 2024 22:10:40 GMT

I was looking for the same info and found that great link below.

https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html

I hope it can help you.

Best,