question Re: Can we use sqoop to Transfer file from a HDFS directory to a remote server. in Support Questions

Can we use sqoop to Transfer file from a HDFS directory to a remote server.

s198 — Wed, 03 Apr 2024 10:07:29 GMT

We have a requirement to transfer files from an HDFS directory to a remote server.

I've noticed options to copy files from HDFS to the local filesystem first (using copyToLocal) and then transfer files from the local filesystem to the remote server (using scp). But is there any direct method to copy files from HDFS to a remote server, such as using Sqoop functionality or any other methods like without copying to the local file system

Re: Can we use sqoop to Transfer file from a HDFS directory to a remote server.

jAnshula — Wed, 03 Apr 2024 10:11:47 GMT

Hi @s198

You can use DistCp command to achieve the same.

refer ---> https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html

Re: Can we use sqoop to Transfer file from a HDFS directory to a remote server.

s198 — Wed, 03 Apr 2024 10:41:38 GMT

Thanks @jAnshula for your suggestion

My remote server does not support Hadoop-compatible file systems, so the DistCp command does not work for me. The primary objective is to copy the HDFS files as is to a Linux machine.

Re: Can we use sqoop to Transfer file from a HDFS directory to a remote server.

jAnshula — Wed, 03 Apr 2024 11:23:23 GMT

@s198

You may try and use NIFI, refer below article

https://community.cloudera.com/t5/Support-Questions/How-to-copy-files-from-a-remote-linux-box-using-Nifi/m-p/145133

Re: Can we use sqoop to Transfer file from a HDFS directory to a remote server.

s198 — Wed, 03 Apr 2024 12:13:50 GMT

Thanks @jAnshula for your suggestion.
We are trying to achieve this without using Nifi. Could you please let us know if any options are available using hdfs/hadoop commands?

Re: Can we use sqoop to Transfer file from a HDFS directory to a remote server.

Majeti — Wed, 03 Apr 2024 16:31:11 GMT

Hi @s198, You do not need to have hadoop file system or datanode role on the remote server. You just need to set up some hdfs gateway on the remote server and pull it using distcp. If you are using HDP or CDP, you can add the remote server as a gateway and perform distcp in the remote server.

Another option is to share one of the directories in the remote server, mount it in hadoop cluster node, and perform distcp to that mounted directory.