Skip to main content
rvillanueva
rvillanueva
Gold ⭐️
July 25, 2019
Solved

Ambari files view upload error: java.net.NoRouteToHostException: myhost:50075: No route to host (Host unreachable)

Trying to upload a simple csv file into HDFS via the Ambari files view, getting the error below

java.net.NoRouteToHostException: hw04.ucera.local:50075: No route to host (Host unreachable)     
at java.net.PlainSocketImpl.socketConnect(Native Method)    
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)    
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)    
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)    
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)    
at java.net.Socket.connect(Socket....

For reference, here the hosts on the cluster

110039-1564016464453.png

and the services on the HW04 node

  • Accumulo Client
  • DataNode
  • HBase Client
  • HDFS Client
  • Hive Client
  • HST Agent
  • Infra Solr Client
  • Log Feeder
  • MapReduce2 Client
  • Metrics Monitor
  • Oozie Client
  • Pig Client
  • Spark2 Client
  • Sqoop Client
  • Tez Client
  • YARN Client
  • ZooKeeper Client

Not sure what the error means. Any debugging suggestions or fixes?


1564016455599.png
rvillanueva
rvillanuevaAnswer
Gold ⭐️
July 27, 2019

Think I found the problem, TLDR: firewalld (nodes running on centos7) was still running, when should be disabled on HDP clusters.

From another community post:

For Ambari to communicate during setup with the hosts it deploys to and manages, certain ports must be open and available. The easiest way to do this is to temporarily disable iptables, as follows:

systemctl disable firewalld

service firewalld stop

So apparently iptables and firewalld need to be disabled across the cluster (supporting docs can be found here, I only disabled them on the Ambari installation node). After stopping these services across the cluster (I recommend using clush), was able to run the upload job without incident.

View in thread

7 replies

jsensharma
Platinum ⭐️⭐️
jsensharma
Platinum ⭐️⭐️
July 25, 2019

@Reed Villanueva

The most important requirement for the Cluster is that Not only ambari to be able to resolve each cluster nodes with they FQDN.

BUT all the cluster nodes also should be able to resolve each other using their FQDN.

So please make sure that all your cluster nodes are able to resolve each other using their hostname / FQDN


Please check the "/etc/hosts" file on ambari server node as well as on other cluster nodes. (they can be same to resolve all the cluster nodes). If you are using DNS entry then please make sure that each host is able to resolve each others DNS name.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation-ppc/content/edit_the_host_file.html
https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-installation-ppc/content/check_dns.html

Translate
jsensharma
Platinum ⭐️⭐️
jsensharma
Platinum ⭐️⭐️
July 25, 2019

@Reed Villanueva

Please check if you can acccess the hostname "hw04.ucera.local" from all your cluster nodes? Without any hostname/firewall issue?


Please run the same command from all cluster nodes:

# ping hw04.ucera.local
# telnet hw04.ucera.local  50075

# hostname -f

.

Translate
rvillanueva
rvillanuevaAuthor
Gold ⭐️
July 25, 2019

@Jay Kumar SenSharma

Checking the /etc/hosts files on the hw[1-3] cluster nodes, there was no hw04 FQDN (note that these were the original nodes, hw04 was a node that was added later via Ambari), have added now as well as to hw04. But was able to ping hw04 from all the other nodes regardless. However, was not able to run

telnet hw04.ucera.local  50075

from any of the nodes. Not sure what this means or how to fix it, though?


** Also interesting that Ambari does not automatically added the new FQDN to the /etc/hosts file of the existing nodes when added another node (since already has root access). So seems that this means that any time an admin wanted to add more nodes, they would need to go through all the cluster nodes and add this manually. Any specific reason for this design?

Translate
rvillanueva
rvillanuevaAuthorAnswer
Gold ⭐️
July 27, 2019

Think I found the problem, TLDR: firewalld (nodes running on centos7) was still running, when should be disabled on HDP clusters.

From another community post:

For Ambari to communicate during setup with the hosts it deploys to and manages, certain ports must be open and available. The easiest way to do this is to temporarily disable iptables, as follows:

systemctl disable firewalld

service firewalld stop

So apparently iptables and firewalld need to be disabled across the cluster (supporting docs can be found here, I only disabled them on the Ambari installation node). After stopping these services across the cluster (I recommend using clush), was able to run the upload job without incident.

Translate
Terms & ConditionsAccessibility statement