Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Distcp vs hdfs cp

Solved Go to solution

Distcp vs hdfs cp

New Contributor

Hello All,

I have a requirement where i want to copy files from one hdfs directory to another via oozie in same cluster.

This can be done using oozie discp action or oozie shell action.

Which is a better way to copy files using oozie.

I guess it is similar as asking hdfs -cp vs distcp?

Thanks and Best Regards,

Gagan

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Distcp vs hdfs cp

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

3 REPLIES 3

Re: Distcp vs hdfs cp

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

Re: Distcp vs hdfs cp

New Contributor

This is very much the same i researched too. So i go with distcp for my usecase.

Re: Distcp vs hdfs cp

Rising Star

All,

Just adding for knowledge gain , if my source is kerberos enabled while target is not , then the command to be executed will be

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://source-ip webhdfs://target-ip

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true : this command overrides values present in the hive-site.xml.

thanks,

Rishit Shah