Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

hadoop distcp failing with Invalid arguments: ipoaddres:9870: Connection refused

avatar
New Contributor

I am trying to copy files from CDH5.16 to CDH6 using the following command

 

               hadoop distcp webhdfs://10.0.0.154:50070/cc/cc_apiused.csv webhdfs://10.0.2.131/cc/

 

and I am getting the following error:

 

                ERROR tools.DistCp: Invalid arguments: java.net.ConnectException: ip-10-0-2-131.us-west-2.compute.internal:9870: Connection refused (Connection refused)

 

From Cloudera Manager UI Cluster is healthy. Attached image CDHHea.th.jpg for your reference.

 

I have 7 EC2 instances on AWS

 

Cloudera Manager and a databaes on one server -- CMserver 10.0.0.238

 

from the Cloudera Manager I build the cluster using 6 EC2 instances 

NameNode   - 1 instance 10.0.2.131

Secondary Name  - 1 instance  10.0.2.46

DataNodes - 4 instances 10.0.2.[27,81,23,200]

 

CDH roles are as follows

 

ip-10-0-2-131.us-west-2.compute.internal1
 B  NN  G  HMS  HS2  AP  ES  HM  RM  SM  G  JHS  RM  S
ip-10-0-2-27.us-west-2.compute.internal1
 DN  G  OS  G  NM
ip-10-0-2-46.us-west-2.compute.internal1
 SNN  G  HS2  G  HS  S
ip-10-0-2-81.us-west-2.compute.internal1
 DN  G  G  NM  S
ip-10-0-2-[23, 200].us-west-2.compute.internal2
 DN  G  G  NM

 

 

Any help to resolve this issue is helpful.

 

Thanks,

--Sriram


CDHHealth.JPG
1 REPLY 1

avatar
Master Collaborator

Hello @kotni

 

The Stack trace says the connection is refused on the NameNode port(9870)dfs.namenode.http-address, hence make sure the required ports are configured/opened in both the source and Destination clusters on performing Distcp.

 

- Here is the list of ports used by CDH components and DistCP

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_ports.html

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/install_ports_distcp.html

 

Post that, Try out the below:

- Examples of DistCP syntaxes to try out

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_admin_distcp_data_cluster_migrat...

 

Let us know how it goes.

- Vaishaak