Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Failed to send a small file across remote cluster via distcp command?

Failed to send a small file across remote cluster via distcp command?

I am using distcp command to send data from one cluster to another remote cluster but failed to do so.Getting error
'file could only be written to 0 of the 1 minreplication (=1) nodes".

I got stuck into it please do suggest me the solution.
Thanks in advance.

2 REPLIES 2

Re: Failed to send a small file across remote cluster via distcp command?

Mentor

@Muhammad waqas

Can you share your DISTCP command?

Make sure the name node are up and running!

Check if the data nodes are in the dfs.exclude

$cat /etc/hadoop/conf/dfs.exclude

It seems your replication factor is higher than the number of data nodes! Can you manually set the replication factor to 1 and retry .

$ hadoop distcp -D dfs.replication=1 hdfs://namenode1:50070/file1  hdfs://namenode2:50070/

Hope that helps


Highlighted

Re: Failed to send a small file across remote cluster via distcp command?

@Geoffrey Shelton Okot

Thanks for the reply.

Actually i am trying to send data from hortonworks to cloudera cluster.

dfs.exclude file is empty.

The distcp command you've written has some issues,we can't attach 50070 port with hdfs prefix to do this write webhdfs instead of hdfs.

my distcp command is
hadoop --config (path of directory containing hdfs-site or core-site.xml files of targeting cluster) distcp hdfs://nn1/path hdfs://nn2/path

Aforementioned command can send data from cloudera to hortonworks cluster but i want to do it in reverse direction.