Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

distcp not distributing

avatar
Contributor

I'm transferring files using distcp on Cloudera 5.3.x, and I can't get it to distribute the transfer using MR2.

 

I don't have MR1 installed at all. I'd rather not because it'll hide issues. 

 

My command line looks like this, and it runs just fine, it just copies every file in series:

 

mapred distcp s3n://key:secret@logs.space.com/source/2015/04/28/ hdfs://nameservice/target/dir/2015/04/28

 

Is there a configuration item that I missed?

1 ACCEPTED SOLUTION

avatar
Mentor
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

View solution in original post

2 REPLIES 2

avatar
Mentor
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

avatar
Contributor

That was it! Thank you!