hadoop distcp failing with Invalid arguments: ipoaddres:9870: Connection refused

[#1]

I am trying to copy files from CDH5.16 to CDH6 using the following command

               hadoop distcp webhdfs://10.0.0.154:50070/cc/cc_apiused.csv webhdfs://10.0.2.131/cc/

and I am getting the following error:

                ERROR tools.DistCp: Invalid arguments: java.net.ConnectException: ip-10-0-2-131.us-west-2.compute.internal:9870: Connection refused (Connection refused)

From Cloudera Manager UI Cluster is healthy. Attached image CDHHea.th.jpg for your reference.

I have 7 EC2 instances on AWS

Cloudera Manager and a databaes on one server -- CMserver 10.0.0.238

from the Cloudera Manager I build the cluster using 6 EC2 instances 

NameNode   - 1 instance 10.0.2.131

Secondary Name  - 1 instance  10.0.2.46

DataNodes - 4 instances 10.0.2.[27,81,23,200]

CDH roles are as follows

ip-10-0-2-131.us-west-2.compute.internal	1	B  NN  G  HMS  HS2  AP  ES  HM  RM  SM  G  JHS  RM  S
ip-10-0-2-27.us-west-2.compute.internal	1	DN  G  OS  G  NM
ip-10-0-2-46.us-west-2.compute.internal	1	SNN  G  HS2  G  HS  S
ip-10-0-2-81.us-west-2.compute.internal	1	DN  G  G  NM  S
ip-10-0-2-[23, 200].us-west-2.compute.internal	2	DN  G  G  NM

Any help to resolve this issue is helpful.

Thanks,

--Sriram

[#2]

Hello @kotni, 

The Stack trace says the connection is refused on the NameNode port(9870)dfs.namenode.http-address, hence make sure the required ports are configured/opened in both the source and Destination clusters on performing Distcp.

- Here is the list of ports used by CDH components and DistCP

- https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_ports.html

- https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/install_ports_distcp.html

Post that, Try out the below:

- Examples of DistCP syntaxes to try out

https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cdh_admin_distcp_data_cluster_migrat...

Let us know how it goes.

- Vaishaak