Support Questions
Find answers, ask questions, and share your expertise

DataXceiver error processing READ_BLOCK operation

Rising Star

Frequently getting these below error messages on datanode

ERROR datanode.DataNode (DataXceiver.java:run(250)) - X.X.X.X6:50010:DataXceiver error processing READ_BLOCK operation src: /x.x.x.7:49636 dst: /x.x.x.6:50010 java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.6:50010 remote=/x.x.x.7:49636] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:172) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:220) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:547) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:716) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:506) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:110) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232) at java.lang.Thread.run(Thread.java:745)

8 REPLIES 8

Mentor

@Anshul Sisodia check your networking/firewall on this node.

Mentor

Rising Star

@Artem Ervits no networking or firewall issue as such

@Anshul Sisodia - It looks like you have a connection issue.

a) Check on the destination host if the datanode is up and running

b) you can run tcpdump between the two hosts and port and monitor the traffic

https://danielmiessler.com/study/tcpdump/

tcpdump is an excellent tool that can give you lot of network related problem information.

Rising Star

a) Datanodes are up and running.

b) Will check the tcpdump and send the output

The stack trace indicates the DataNode was serving a client block read operation. It attempted to write some data to the client on the socket connection, but the write timed out. This likely indicates a client-side problem, not a DataNode problem. I agree with the assessments to check networking. If you know what client application was running from the "remote" address in the stack trace, then it's also helpful to investigate any logs generated by that application.

Mentor

@Anshul Sisodia are you still having problems with this? Can you provide your own solution or accept best answer?