Not able to start service

[#1]

ambari.pngambari-1.pngWe are facing this problem since last week and aren't able to start services. We are trying to install HDP(2.4) on a 3 node cluster. We were able to download and install all the packages via ambari GUI but it failed to start the services in the last step of installation. Then I tried to start service manually. Some of the services are started. While checking we found below error. Can any one tell me the issue?

Errno 111] Connection refused to XXX.yyy.com:21000 Connection failed to http://XXX.yyy.com:21000 (urlopen error [Errno 111] Connection refused)

Connection failed to http://XXX.yyy.com:50070 (urlopen error [Errno 111]

Connection refused) Connection failed to http://XXX.yyy.com:8088 ( )

All hosts are pingable from each other in the cluster. .There are entry in /etc/host file.

While checking http://<hostname>:50070 , it gives me below error

Connection refused

The server does not have a DNS entry. Perhaps there is a misspelling in the server name, or the server no longer exists. Double-check the name and try again.

[#2]

@Rajib Mandal

Can you describe your network setup ??ie Firewall rules bwteen these 3 nodes ?

[#3]

All hosts are pingable from each other in the cluster. There are entry in /etc/host file.

Sample host file is given below

127.0.0.1localhost

127.0.1.1abc.yyy.COM abc00675972001

101.323.70.140 abc00675972001.yyy.COM

101.323.71.166 abc00675972002.yyy.COM

101.323.71.165 abc00675972003.yyy.COM

# The following lines are desirable for IPv6 capable hosts

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

[#4]

@Rajib Mandal

It looks that the port 50070 is listening on loopback IP , netstat -ntlup command shows that.

Please make sure you have "dfs.namenode.http-address" property in hdfs-site.xml set to the <HostIPAddress>:50070

This can be set from Ambar Dashboard>Services>HDFS>Configs>Advanced>Advanced hdfs-site

Look for property dfs.namenode.http-address and set it to <HostIPAddress>:50070

Restart the hdfs for this to take effect. It should look something like below

[root@hdp1 ~]# netstat -an | grep 50070
tcp 0 0 192.168.56.10:50070 0.0.0.0:* LISTEN    <<<<< 
tcp 0 0 192.168.56.10:50070 192.168.56.1:51147 ESTABLISHED
tcp 0 0 192.168.56.10:50070 192.168.56.1:51144 ESTABLISHED

Instead of this (which is in your case now)

[root@hdp1 ~]# netstat -an | grep 50070 
tcp 0 0 127.0.0.1:50070 0.0.0.0:* LISTEN 
tcp 1 0 ::ffff:192.168.56.10:55036 ::ffff:192.168.56.10:50070 CLOSE_WAIT 
tcp 1 0 ::ffff:192.168.56.10:55094 ::ffff:192.168.56.10:50070 CLOSE_WAIT

[#5]

I was facing the same issue. It looks like it is related to hostname.

In dfs.namenode.http properties I have added IP address instead of hostname.

Which resolved my issue.