Support Questions

Find answers, ask questions, and share your expertise

Distcp vs hdfs cp

avatar
Explorer

Hello All,

I have a requirement where i want to copy files from one hdfs directory to another via oozie in same cluster.

This can be done using oozie discp action or oozie shell action.

Which is a better way to copy files using oozie.

I guess it is similar as asking hdfs -cp vs distcp?

Thanks and Best Regards,

Gagan

1 ACCEPTED SOLUTION

avatar
Super Guru

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

View solution in original post

3 REPLIES 3

avatar
Super Guru

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

avatar
Explorer

This is very much the same i researched too. So i go with distcp for my usecase.

avatar
Expert Contributor

All,

Just adding for knowledge gain , if my source is kerberos enabled while target is not , then the command to be executed will be

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://source-ip webhdfs://target-ip

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true : this command overrides values present in the hive-site.xml.

thanks,

Rishit Shah