question Re: Datanode starts but doesn't connect to namenode in Support Questions

Datanode starts but doesn't connect to namenode

iit2013121 — Wed, 10 Feb 2016 02:09:41 GMT

I installed a 2 node hadoop cluster. The master and slave node starts separately but the datanode isn't shown in namenode webUI. The log file for datanode shows the following error :

	2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
	2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
	2016-02-09 23:30:54,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
	2016-02-09 23:30:55,977 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 23:21:15,062 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/172.17.25.5:9000

The hosts file in slave is

172.17.25.5    hadoop-master
127.0.0.1    silp-ProLiant-DL360-Gen9
172.17.25.18    hadoop-slave-1
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

the core-site.xml file is :

<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://hadoop-master:9000</value>
</property>
<property>
    <name>dfs.permissions</name>
       <value>false</value>
</property>
</configuration>

Kindly help with this issue.

Re: Datanode starts but doesn't connect to namenode

nsabharwal — Wed, 10 Feb 2016 02:13:28 GMT

@Kumar Sanyam

Is there connectivity between the servers?

WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Problem connecting to server: hadoop-master/172.17.25.5:9000

Please make sure that ssh works and iptable is off

Re: Datanode starts but doesn't connect to namenode

aervits — Wed, 10 Feb 2016 02:16:19 GMT

@Kumar Sanyam

turn off ipv6 in /etc/hosts.

Re: Datanode starts but doesn't connect to namenode

Shelton — Wed, 10 Feb 2016 03:28:14 GMT

1. To add on to Artem's idea.

Disable IPv6 Temporarily 'echo 1 > /proc/sys/net/ipv6/conf/<interface-name>/disable_ipv6'

eg 'echo 1 > /proc/sys/net/ipv6/conf/eth0/disable_ipv6'

Disable IPv6 Permanently across reboots

Edit the /etc/sysctl.conf and the following lines.

# to disable IPv6 on all interfaces system wide net.ipv6.conf.all.disable_ipv6 = 1

# to disable IPv6 on a specific interface (e.g., eth0, lo)

net.ipv6.conf.lo.disable_ipv6 = 1

net.ipv6.conf.eth0.disable_ipv6 = 1

2. Check the NTP server configuration or just restart NTP on both servers

Re: Datanode starts but doesn't connect to namenode

ashneesharma88 — Wed, 17 Feb 2016 20:21:09 GMT

turn off ipv6 in /etc/sysconfig/network or /etc/sysconfif/network-manager/ifcfg-eth0

parameter

NETWORKING_IPV6=no

IPV6INIT=no

Please check selniux is disable or not if not then disable it and reboot.

vi /etc/sysconfig/selinux

selinux=disabled

Also check ntp and iptable and firewall.

Re: Datanode starts but doesn't connect to namenode

iit2013121 — Thu, 18 Feb 2016 04:05:01 GMT

thanks everyone..solved the problem. Actually the namenode was listening at localhost:9000 and datanode tried to connect at hadoop-master:9000, hence the error in connectivity. Changed the listening IP:port of namenode to hadoop-master:9000.

Re: Datanode starts but doesn't connect to namenode

pradeepkumart — Mon, 18 Jul 2016 13:35:32 GMT

I am facing the same issue. Could you please guide me how did you solve the problem. I am not able to get a clue from the answer.

Re: Datanode starts but doesn't connect to namenode

fcher061 — Wed, 10 Aug 2016 00:34:44 GMT

Would you please tell us how did you change localhost:9000 to master:9000. in core-site it is addressing master:9000. Why it picks 127.0.0.1? Which file I should make changes?

Re: Datanode starts but doesn't connect to namenode

fcher061 — Wed, 10 Aug 2016 00:59:44 GMT

Solution worked for me -> firewall was the reason on 64bit Ubunu 16.04 "sudo ufw allow 9000" changes the rules, then namenode listens with master:9000

Re: Datanode starts but doesn't connect to namenode

fcher061 — Wed, 10 Aug 2016 01:02:26 GMT

You can check the ports with netstat -anp | grep 9000 before and after the change in the ufw

Re: Datanode starts but doesn't connect to namenode

dkant007 — Fri, 14 Dec 2018 03:36:35 GMT

Thanks all for contributing to this post. I had the same issue on Centos 7 on a 2 node hadoop cluster. In my case firewall was the cause. After allowing my namenode port (in my case 8020) through firewall on namenode machine allowed datanodes to connect to namenode.

$ sudo firewall-cmd --zone=public --add-port=8020/tcp --permanent
$ sudo firewall-cmd --reload

Re: Datanode starts but doesn't connect to namenode

ha_ahmadkhani — Sat, 04 May 2019 21:07:14 GMT

add port to firewall and enjoy

sudo firewall-cmd --zone=public --add-port=8020/tcp --permanent
sudo firewall-cmd --reload

Re: Datanode starts but doesn't connect to namenode

Anmol — Wed, 20 Oct 2021 02:22:15 GMT

Thanks for the Solution