Support Questions
Find answers, ask questions, and share your expertise

Distcp vs hdfs cp

Solved Go to solution
Highlighted

Distcp vs hdfs cp

Explorer

Hello All,

I have a requirement where i want to copy files from one hdfs directory to another via oozie in same cluster.

This can be done using oozie discp action or oozie shell action.

Which is a better way to copy files using oozie.

I guess it is similar as asking hdfs -cp vs distcp?

Thanks and Best Regards,

Gagan

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Distcp vs hdfs cp

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

View solution in original post

3 REPLIES 3
Highlighted

Re: Distcp vs hdfs cp

@Gagandeep Singh Chawla,

The usage depends on your use case.

1)The main disadvantage of fs -cp is that all data has to transit via the machine you issue the command on, depending on the size of data you want to copy the time consumed increases. DistCp is distributed as its name implies, so there is no bottleneck of this kind.

2) distcp runs a MR job behind and cp command just invokes the FileSystem copy command for every file.

3) If there are existing jobs running, then distcp might take time depending memory/resources consumed by already running jobs.In this case cp would be better.

4) Also, distcp will work between 2 clusters.

Thanks,

Aditya

View solution in original post

Highlighted

Re: Distcp vs hdfs cp

Explorer

This is very much the same i researched too. So i go with distcp for my usecase.

Highlighted

Re: Distcp vs hdfs cp

Rising Star

All,

Just adding for knowledge gain , if my source is kerberos enabled while target is not , then the command to be executed will be

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true webhdfs://source-ip webhdfs://target-ip

hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true : this command overrides values present in the hive-site.xml.

thanks,

Rishit Shah