Skip to main content
mike_bronson7
mike_bronson7
Gold ⭐️⭐️⭐️
July 7, 2020
Solved

how to copy file from remote server to HDFS

 
 

I have a remote server and servers authenticated Hadoop environment.

I want to copy file from Remote server to Hadoop machine to HDFS

Please advise efficient approach/HDFS command to copy files from remote server to HDFS.

Any example will be helpful.

as ordinary way to copy file from remote server to server itself is

scp -rp file remote_server:/tmp

but this approach not support copy directly to hdfs

vsundaram
vsundaramAnswer
Gold ⭐️
July 14, 2020

@mike_bronson7 Are you trying to copy local files from your remote machine to destination HDFS cluster? You could use distcp if it is between hdfs. Please refer below documentation.

https://docs.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_data_cluster_migrate.html

 

For the local files in remote machine, you could SCP the files to any of the cluster node which has hdfs client installed and the do a "-copyFromLocal" or "-put" to push that to HDFS. Hope this helps.

View in thread

3 replies

vsundaram
Gold ⭐️
vsundaramAnswer
Gold ⭐️
July 14, 2020

@mike_bronson7 Are you trying to copy local files from your remote machine to destination HDFS cluster? You could use distcp if it is between hdfs. Please refer below documentation.

https://docs.cloudera.com/documentation/enterprise/5-5-x/topics/cdh_admin_distcp_data_cluster_migrate.html

 

For the local files in remote machine, you could SCP the files to any of the cluster node which has hdfs client installed and the do a "-copyFromLocal" or "-put" to push that to HDFS. Hope this helps.

Translate
    SujaP
    SujaP
    Silver ⭐️
    December 17, 2021

    Can you please give an example for this?

     

    For the local files in remote machine, you could SCP the files to any of the cluster node which has hdfs client installed and the do a "-copyFromLocal" or "-put" to push that to HDFS. Hope this helps.

     

    Thanks

     

    Translate
    BrunoAndrade
    BrunoAndrade
    Silver ⭐️
    April 17, 2024

    I was looking for the same info and found that great link below.

    https://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html

    I hope it can help you.

    Best,

    Translate
      Terms & ConditionsAccessibility statement