question Re: Distcp vs hdfs cp in Support Questions

Distcp vs hdfs cp

coolgags — Wed, 10 Jan 2018 01:01:31 GMT

Hello All,

I have a requirement where i want to copy files from one hdfs directory to another via oozie in same cluster.

This can be done using oozie discp action or oozie shell action.

Which is a better way to copy files using oozie.

I guess it is similar as asking hdfs -cp vs distcp?

Thanks and Best Regards,

Gagan

Re: Distcp vs hdfs cp

asirna — Wed, 10 Jan 2018 01:22:09 GMT

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

Re: Distcp vs hdfs cp

coolgags — Wed, 10 Jan 2018 16:35:07 GMT

This is very much the same i researched too. So i go with distcp for my usecase.

Re: Distcp vs hdfs cp

rishit606 — Wed, 10 Jan 2018 17:19:50 GMT

All,

Just adding for knowledge gain , if my source is kerberos enabled while target is not , then the command to be executed will be

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://source-ip webhdfs://target-ip

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true : this command overrides values present in the hive-site.xml.

thanks,

Rishit Shah