Posts: 17
Registered: ‎06-02-2014
Accepted Solution

distcp not distributing

I'm transferring files using distcp on Cloudera 5.3.x, and I can't get it to distribute the transfer using MR2.


I don't have MR1 installed at all. I'd rather not because it'll hide issues. 


My command line looks like this, and it runs just fine, it just copies every file in series:


mapred distcp s3n:// hdfs://nameservice/target/dir/2015/04/28


Is there a configuration item that I missed?

Posts: 1,892
Kudos: 431
Solutions: 302
Registered: ‎07-31-2013

Re: distcp not distributing

Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

Posts: 17
Registered: ‎06-02-2014

Re: distcp not distributing

That was it! Thank you!