DataXceiver error processing READ_BLOCK operation

[#1]

Frequently getting these below error messages on datanode

ERROR datanode.DataNode (DataXceiver.java:run(250)) - X.X.X.X6:50010:DataXceiver error processing READ_BLOCK operation src: /x.x.x.7:49636 dst: /x.x.x.6:50010 java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.6:50010 remote=/x.x.x.7:49636] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:172) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:220) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:547) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:716) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:506) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:110) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232) at java.lang.Thread.run(Thread.java:745)

[#2]

@Anshul Sisodia check your networking/firewall on this node.

[#3]

@Chris Nauroth

[#4]

@Artem Ervits no networking or firewall issue as such

[#5]

@Anshul Sisodia - It looks like you have a connection issue.

a) Check on the destination host if the datanode is up and running

b) you can run tcpdump between the two hosts and port and monitor the traffic

https://danielmiessler.com/study/tcpdump/

tcpdump is an excellent tool that can give you lot of network related problem information.

[#6]

a) Datanodes are up and running.

b) Will check the tcpdump and send the output

[#7]

The stack trace indicates the DataNode was serving a client block read operation. It attempted to write some data to the client on the socket connection, but the write timed out. This likely indicates a client-side problem, not a DataNode problem. I agree with the assessments to check networking. If you know what client application was running from the "remote" address in the stack trace, then it's also helpful to investigate any logs generated by that application.

[#8]

@Anshul Sisodia

I hope you are not hitting this https://issues.apache.org/jira/browse/HDFS-770

[#9]

@Anshul Sisodia are you still having problems with this? Can you provide your own solution or accept best answer?