Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

distcp not distributing

Solved Go to solution

distcp not distributing

Explorer

I'm transferring files using distcp on Cloudera 5.3.x, and I can't get it to distribute the transfer using MR2.

 

I don't have MR1 installed at all. I'd rather not because it'll hide issues. 

 

My command line looks like this, and it runs just fine, it just copies every file in series:

 

mapred distcp s3n://key:secret@logs.space.com/source/2015/04/28/ hdfs://nameservice/target/dir/2015/04/28

 

Is there a configuration item that I missed?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: distcp not distributing

Master Guru
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

2 REPLIES 2

Re: distcp not distributing

Master Guru
Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

Re: distcp not distributing

Explorer

That was it! Thank you!