question Failed to find datanode, suggest to check cluster health. excludeDatanodes=null in Archives of Support Questions (Read Only)

Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Thu, 26 Oct 2017 22:01:33 GMT

Hello guys,

I have setup a cluster and HDFS is up and running.

But MapReduce2 and Spark not getting started. It gives the below error.

{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "Failed to find datanode, suggest to check cluster health. excludeDatanodes=null"
  }
}

Hbase is also getting the below error

org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /apps/hbase/data/.tmp/hbase.version could only be replicated to 0 nodes instead of minReplication (=1).There are 0 datanode(s) running and no node(s) are excluded in this operation.

Somehow the datanodes are not detected.

I updated the hosts file in all nodes with the private ip and private DNS. But still getting the same error.

Can someone please help me in this.

Thanks,

Nirmal J

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

elserj — Thu, 26 Oct 2017 22:44:35 GMT

It looks like HDFS is not actually running. The message you copied from HBase states that there are no datanodes running.

Look at the DataNode logs and check the NameNode WebUI to try to identify why the DataNodes aren't running/included.

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Thu, 26 Oct 2017 23:57:11 GMT

Thanks @Josh Elser,

I see the below error in the datanode logs

2017-10-26 16:53:28,746 INFOipc.Server (Server.java:run(900)) - IPC Server listener on 8010: starting

2017-10-26 16:53:29,892 INFOipc.Client (Client.java:handleConnectionFailure(906)) - Retrying connect to server: localhost/127.0.0.1:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

Is it because the namenode IP is specified wrong in some config?

Thanks,

Nirmal J

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

asirna — Fri, 27 Oct 2017 00:32:26 GMT

@Nirmal J,

From the above log looks this there is an issue with /etc/hosts configuration. Can you please try

telnet <namenode-host> 8020

The datanode is pointing to localhost/127.0.0.1:8020 instead of the name node host.

Thanks,

Aditya

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Fri, 27 Oct 2017 00:39:08 GMT

@Aditya Sirna Thanks for your time.

telnet gives connection refused error

But the service is running on port 8020 in namenode

Here is my master node etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost6 localhost6.localdomain6

10.0.65.59 ip-10-0-65-59.ec2.internal ip-10-0-65-59

10.0.125.23 ip-10-0-125-23.ec2.internal

10.0.190.115 ip-10-0-190-115.ec2.internal

And this is my slave node /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost6 localhost6.localdomain6

10.0.125.23 ip-10-0-125-23.ec2.internal ip-10-0-125-23

10.0.65.59 ip-10-0-65-59.ec2.internal

10.0.190.115 ip-10-0-190-115.ec2.internal

Thanks,

Nirmal J

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

asirna — Fri, 27 Oct 2017 00:53:12 GMT

@Nirmal J

Can you please check the value of dfs.namenode.rpc-address from Advanced hdfs-site.xml

Thanks, Aditya

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Fri, 27 Oct 2017 00:55:33 GMT

@Aditya Sirna

It is localhost:8020

Thanks,

Nirmal J

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

asirna — Fri, 27 Oct 2017 00:56:57 GMT

Please change it to hostname and try restarting the services. Make sure that "telnet hostname 8020" works

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Fri, 27 Oct 2017 11:07:55 GMT

@Aditya Sirna I tried to give the namenode public IP there. But the namenode itself failed to start

Call From ip-10-0-65-59.ec2.internal/10.0.65.59 to 34.231.98.121:8020 failed on connection exception: java.net.ConnectException: Connection refused;

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

asirna — Fri, 27 Oct 2017 11:19:11 GMT

@Nirmal J,

I would recommend using hostnames than IPs. If you want to use IPs instead , use either all private IPs or all public. As you have configured all private IPs in /etc/hosts , it is better to use private IPs. But my recommendation is to go with hostnames.

Also , make sure that you able to ping with the private IPs and hostnames from one box to another

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Fri, 27 Oct 2017 14:23:23 GMT

@Aditya Sirna The error seems to be weird

1) The namenode service is up and running in the master

[cloudbreak@ip-10-0-223-116 ~]$ sudo netstat -nltp| grep :8020

tcp00 127.0.0.1:80200.0.0.0:* LISTEN6689/java

2) Port 8020 is open for allscreen-shot-2017-10-27-at-124602-pm.png

3) I can ping from master to slave using the private DNS

4) But telnet is getting connection refused

[cloudbreak@ip-10-0-109-247 hdfs]$ telnet 10.0.223.116 8020

Trying 10.0.223.116...

telnet: connect to address 10.0.223.116: Connection refused

5) Datanode logs has the below error

2017-10-27 05:24:15,073 INFOipc.Client (Client.java:handleConnectionFailure(906)) - Retrying connect to server: ip-10-0-223-116.ec2.internal/10.0.223.116:8020. Already tried 48 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

asirna — Fri, 27 Oct 2017 14:56:57 GMT

@Nirmal J,

Can you set dfs.namenode.rpc-address to ip-10-0-223-116 and restart the namenode.

Hope you have the below mapping in /etc/hosts

10.0.223.116 ip-10-0-223-116

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

jsensharma — Fri, 27 Oct 2017 15:07:22 GMT

@Nirmal J

In one of your previous update you mentioned that "I can ping from master to slave using the private DNS, But telnet is getting connection refused"

That can happen if there is a firewall issue. So can you also please check if the "iptables" (or firewalld) is stopped on NameNode host and also on all the datanode hosts. Please share the output of the following from the DataNode host.

# service iptables status
# service iptables stop
# ping  telnet 10.0.223.116 
# ping  ip-10-0-223-116
# telnet 10.0.223.116 8020

Also the "/etc/hosts" file accross the cluster should look almost idnenticle so that every node of the cluster can connect/resolve to each other using their FQDN/Hostname.

Also please check of the "hostname -f" is returning the correct FQDN on every host Specially the NameNode and on the DataNode that is failing to connect.

# hostname -f

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

jsensharma — Fri, 27 Oct 2017 15:12:39 GMT

@Nirmal J

Also based on your previous shared "/etc/hosts" file content i do not see any where the following address is mapped:

10.0.223.116	  ip-10-0-65-59.ec2.internal	ip-10-0-65-59.ec2

Is this your NameNode address ? "34.231.98.121" Or "10.0.223.116"

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

jsensharma — Fri, 27 Oct 2017 15:15:14 GMT

@Nirmal J

Can you also share the output of the following commands from the NameNode host?

# service iptables status
# hostname -f
# netstat -tnlpa | grep `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Fri, 27 Oct 2017 15:29:21 GMT

This didn't work @Aditya Sirna. Still the same issue.

It seems like some kind of firewall issue.

telnet 10.0.223.116 8020 is connecting from namenode, but not from datanode

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

nirmal1 — Fri, 27 Oct 2017 15:32:27 GMT

@Jay SenSharma

[cloudbreak@ip-10-0-223-116 ~]$ sudo service iptables status

iptables: Firewall is not running.

[cloudbreak@ip-10-0-223-116 ~]$ hostname -f

ip-10-0-223-116.ec2.internal

[cloudbreak@ip-10-0-223-116 ~]$ sudo netstat -tnlpa | grep `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid`

tcp00 127.0.0.1:80200.0.0.0:* LISTEN29280/java

tcp00 127.0.0.1:50070 0.0.0.0:* LISTEN29280/java

tcp00 127.0.0.1:20103 0.0.0.0:* LISTEN29280/java

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

jsensharma — Fri, 27 Oct 2017 15:50:58 GMT

@Nirmal J

It is strange to see the address 127.0.0.1:8020 in your output which says that the port 8020 is listening on 127.0.0.1 address means from remote machine that port can not be accessed.

Ideally the NameNode address should be showing something like following:

# netstat -tnlpa | grep `cat /var/run/hadoop/hdfs/hadoop-hdfs-namenode.pid`
tcp        0      0 172.27.10.2:8020             0.0.0.0:*                   LISTEN      509/java            
tcp        0      0 172.27.10.2:50070            0.0.0.0:*                   LISTEN      509/java

Can you please check and share the core-site.xml / hdfs-site.xml to see if th NameNode address is correct and it listening on Hostname (FQDN) and not on 127.0.0.1

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

jsensharma — Fri, 27 Oct 2017 15:54:11 GMT

@Nirmal J

Please check the ifconfig output on the NameNode to see the actual IP Address that it is supposed to be using.

Example:

# ifconfig
eth0  Link encap:Ethernet  HWaddr 12:42:AA:11:00:02  
  inet addr:172.27.10.2  Bcast:0.0.0.0  Mask:255.255.0.0
.
.
lo  Link encap:Local Loopback  
  inet addr:127.0.0.1  Mask:255.0.0.0

Re: Failed to find datanode, suggest to check cluster health. excludeDatanodes=null

jsensharma — Fri, 27 Oct 2017 16:02:42 GMT

@Nirmal J

As your NameNode port 127.0.0.1:8020 indicates that it is listening on "127.0.0.1" IP hence remotely the port 8020 can not be contacted.

So you should check the "dfs.namenode.rpc-address" property of your HDFS to see if it is configured properly with "$HOSTNAME:8020" or not?

And if that hostname is resolving to the IPAddress of the NameNode host?

Can you please check and share the core-site.xml / hdfs-site.xml to see if th NameNode address is correct and it listening on Hostname (FQDN) and not on 127.0.0.1

This can happen normally in two cases.

1. If the /etc/hosts file entry on NameNode is not correct.

2. The other network interface addresses are not resolving.