Distcp vs hdfs cp

[#1]

Hello All,

I have a requirement where i want to copy files from one hdfs directory to another via oozie in same cluster.

This can be done using oozie discp action or oozie shell action.

Which is a better way to copy files using oozie.

I guess it is similar as asking hdfs -cp vs distcp?

Thanks and Best Regards,

Gagan

[#2 · Accepted solution]

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

[#3]

This is very much the same i researched too. So i go with distcp for my usecase.

[#4]

All,

Just adding for knowledge gain , if my source is kerberos enabled while target is not , then the command to be executed will be

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://source-ip webhdfs://target-ip

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true : this command overrides values present in the hive-site.xml.

thanks,

Rishit Shah