question Re: distcp not distributing in Archives of Support Questions (Read Only)

distcp not distributing

Mythobeast — Fri, 16 Sep 2022 09:27:41 GMT

I'm transferring files using distcp on Cloudera 5.3.x, and I can't get it to distribute the transfer using MR2.

I don't have MR1 installed at all. I'd rather not because it'll hide issues.

My command line looks like this, and it runs just fine, it just copies every file in series:

mapred distcp s3n://key:secret@logs.space.com/source/2015/04/28/ hdfs://nameservice/target/dir/2015/04/28

Is there a configuration item that I missed?

Re: distcp not distributing

Harsh J — Fri, 01 May 2015 17:56:13 GMT

Is the host you run the command on carrying a YARN gateway role, i.e.
valid RM configs under /etc/hadoop/conf/yarn-site.xml?

Do you see the word 'LocalJobRunner' in the output logs of the DistCp
command when its running?

Re: distcp not distributing

Mythobeast — Fri, 01 May 2015 21:30:55 GMT

That was it! Thank you!