Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Datanode starts but doesn't connect to namenode

Solved Go to solution

Datanode starts but doesn't connect to namenode

New Contributor

I installed a 2 node hadoop cluster. The master and slave node starts separately but the datanode isn't shown in namenode webUI. The log file for datanode shows the following error :

	2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
	2016-02-09 23:30:53,920 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 50020: starting
	2016-02-09 23:30:54,976 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
	2016-02-09 23:30:55,977 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hadoop-master/172.17.25.5:9000. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2016-02-09 23:21:15,062 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: hadoop-master/172.17.25.5:9000

The hosts file in slave is

172.17.25.5    hadoop-master
127.0.0.1    silp-ProLiant-DL360-Gen9
172.17.25.18    hadoop-slave-1
# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

the core-site.xml file is :

<configuration>
<property>
  <name>fs.default.name</name>
    <value>hdfs://hadoop-master:9000</value>
</property>
<property>
    <name>dfs.permissions</name>
       <value>false</value>
</property>
</configuration>

Kindly help with this issue.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Datanode starts but doesn't connect to namenode

New Contributor

thanks everyone..solved the problem. Actually the namenode was listening at localhost:9000 and datanode tried to connect at hadoop-master:9000, hence the error in connectivity. Changed the listening IP:port of namenode to hadoop-master:9000.

11 REPLIES 11

Re: Datanode starts but doesn't connect to namenode

@Kumar Sanyam

Is there connectivity between the servers?

WARN org.apache.hadoop.hdfs.server.datanode.DataNode:Problem connecting to server: hadoop-master/172.17.25.5:9000

Please make sure that ssh works and iptable is off

Re: Datanode starts but doesn't connect to namenode

Mentor
@Kumar Sanyam

turn off ipv6 in /etc/hosts.

Re: Datanode starts but doesn't connect to namenode

Mentor

1. To add on to Artem's idea.

Disable IPv6 Temporarily 'echo 1 > /proc/sys/net/ipv6/conf/<interface-name>/disable_ipv6'

eg 'echo 1 > /proc/sys/net/ipv6/conf/eth0/disable_ipv6'

Disable IPv6 Permanently across reboots

Edit the /etc/sysctl.conf and the following lines.

# to disable IPv6 on all interfaces system wide net.ipv6.conf.all.disable_ipv6 = 1

# to disable IPv6 on a specific interface (e.g., eth0, lo)

net.ipv6.conf.lo.disable_ipv6 = 1

net.ipv6.conf.eth0.disable_ipv6 = 1

2. Check the NTP server configuration or just restart NTP on both servers

Re: Datanode starts but doesn't connect to namenode

turn off ipv6 in /etc/sysconfig/network or /etc/sysconfif/network-manager/ifcfg-eth0

parameter

NETWORKING_IPV6=no

IPV6INIT=no

Please check selniux is disable or not if not then disable it and reboot.

vi /etc/sysconfig/selinux

selinux=disabled

Also check ntp and iptable and firewall.

Re: Datanode starts but doesn't connect to namenode

New Contributor

thanks everyone..solved the problem. Actually the namenode was listening at localhost:9000 and datanode tried to connect at hadoop-master:9000, hence the error in connectivity. Changed the listening IP:port of namenode to hadoop-master:9000.

Re: Datanode starts but doesn't connect to namenode

New Contributor

I am facing the same issue. Could you please guide me how did you solve the problem. I am not able to get a clue from the answer.

Re: Datanode starts but doesn't connect to namenode

New Contributor

Would you please tell us how did you change localhost:9000 to master:9000. in core-site it is addressing master:9000. Why it picks 127.0.0.1? Which file I should make changes?

Re: Datanode starts but doesn't connect to namenode

New Contributor

Solution worked for me -> firewall was the reason on 64bit Ubunu 16.04 "sudo ufw allow 9000" changes the rules, then namenode listens with master:9000

Re: Datanode starts but doesn't connect to namenode

New Contributor

You can check the ports with netstat -anp | grep 9000 before and after the change in the ufw

Don't have an account?
Coming from Hortonworks? Activate your account here