Reply
Explorer
Posts: 17
Registered: ‎06-02-2014
Accepted Solution

distcp not distributing

I'm transferring files using distcp on Cloudera 5.3.x, and I can't get it to distribute the transfer using MR2.

 

I don't have MR1 installed at all. I'd rather not because it'll hide issues. 

 

My command line looks like this, and it runs just fine, it just copies every file in series:

 

mapred distcp s3n://key:secret@logs.space.com/source/2015/04/28/ hdfs://nameservice/target/dir/2015/04/28

 

Is there a configuration item that I missed?

Highlighted
Posts: 1,903
Kudos: 435
Solutions: 307
Registered: ‎07-31-2013

Re: distcp not distributing

Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

Explorer
Posts: 17
Registered: ‎06-02-2014

Re: distcp not distributing

That was it! Thank you!