Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

distcp not distributing

avatar
Contributor

I'm transferring files using distcp on Cloudera 5.3.x, and I can't get it to distribute the transfer using MR2.

 

I don't have MR1 installed at all. I'd rather not because it'll hide issues. 

 

My command line looks like this, and it runs just fine, it just copies every file in series:

 

mapred distcp s3n://key:secret@logs.space.com/source/2015/04/28/ hdfs://nameservice/target/dir/2015/04/28

 

Is there a configuration item that I missed?

1 ACCEPTED SOLUTION

avatar
Mentor
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

View solution in original post

2 REPLIES 2

avatar
Mentor
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

avatar
Contributor

That was it! Thank you!