Support Questions

Find answers, ask questions, and share your expertise

distcp not distributing

avatar
Contributor

I'm transferring files using distcp on Cloudera 5.3.x, and I can't get it to distribute the transfer using MR2.

 

I don't have MR1 installed at all. I'd rather not because it'll hide issues. 

 

My command line looks like this, and it runs just fine, it just copies every file in series:

 

mapred distcp s3n://key:secret@logs.space.com/source/2015/04/28/ hdfs://nameservice/target/dir/2015/04/28

 

Is there a configuration item that I missed?

1 ACCEPTED SOLUTION

avatar
Mentor
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

View solution in original post

2 REPLIES 2

avatar
Mentor
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

avatar
Contributor

That was it! Thank you!