Resolving HDFS Connectivity: Changing NameNode Bind Address from Localhost to Hostname

[#1]

I installed a 2 node hadoop cluster. The master and slave node starts separately but the datanode isn't shown in namenode webUI. The log file for datanode shows the following error :

	2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
	2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
	2016-02-09 23:30:54,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
	2016-02-09 23:30:55,977 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 23:21:15,062 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/172.17.25.5:9000

The hosts file in slave is

172.17.25.5    hadoop-master
127.0.0.1    silp-ProLiant-DL360-Gen9
172.17.25.18    hadoop-slave-1
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

the core-site.xml file is :

<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://hadoop-master:9000</value>
</property>
<property>
    <name>dfs.permissions</name>
       <value>false</value>
</property>
</configuration>

Kindly help with this issue.

[#2 · Accepted solution]

@Kumar Sanyam

Is there connectivity between the servers?

WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Problem connecting to server: hadoop-master/172.17.25.5:9000

Please make sure that ssh works and iptable is off

[#3 · Accepted solution]

@Kumar Sanyam

Is there connectivity between the servers?

WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Problem connecting to server: hadoop-master/172.17.25.5:9000

Please make sure that ssh works and iptable is off

[#4]

@Kumar Sanyam

turn off ipv6 in /etc/hosts.

[#5]

1. To add on to Artem's idea.

Disable IPv6 Temporarily 'echo 1 > /proc/sys/net/ipv6/conf/<interface-name>/disable_ipv6'

eg 'echo 1 > /proc/sys/net/ipv6/conf/eth0/disable_ipv6'

Disable IPv6 Permanently across reboots

Edit the /etc/sysctl.conf and the following lines.

# to disable IPv6 on all interfaces system wide net.ipv6.conf.all.disable_ipv6 = 1

# to disable IPv6 on a specific interface (e.g., eth0, lo)

net.ipv6.conf.lo.disable_ipv6 = 1

net.ipv6.conf.eth0.disable_ipv6 = 1

2. Check the NTP server configuration or just restart NTP on both servers

[#6]

turn off ipv6 in /etc/sysconfig/network or /etc/sysconfif/network-manager/ifcfg-eth0

parameter

NETWORKING_IPV6=no

IPV6INIT=no

Please check selniux is disable or not if not then disable it and reboot.

vi /etc/sysconfig/selinux

selinux=disabled

Also check ntp and iptable and firewall.

[#7 · Accepted solution]

thanks everyone..solved the problem. Actually the namenode was listening at localhost:9000 and datanode tried to connect at hadoop-master:9000, hence the error in connectivity. Changed the listening IP:port of namenode to hadoop-master:9000.

[#8]

I am facing the same issue. Could you please guide me how did you solve the problem. I am not able to get a clue from the answer.

[#9]

Would you please tell us how did you change localhost:9000 to master:9000. in core-site it is addressing master:9000. Why it picks 127.0.0.1? Which file I should make changes?

[#10]

Solution worked for me -> firewall was the reason on 64bit Ubunu 16.04 "sudo ufw allow 9000" changes the rules, then namenode listens with master:9000

[#11]

You can check the ports with netstat -anp | grep 9000 before and after the change in the ufw