Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

DataXceiver error processing READ_BLOCK operation

avatar
Expert Contributor

Frequently getting these below error messages on datanode

ERROR datanode.DataNode (DataXceiver.java:run(250)) - X.X.X.X6:50010:DataXceiver error processing READ_BLOCK operation src: /x.x.x.7:49636 dst: /x.x.x.6:50010 java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.6:50010 remote=/x.x.x.7:49636] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:172) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:220) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:547) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:716) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:506) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:110) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232) at java.lang.Thread.run(Thread.java:745)

8 REPLIES 8

avatar
Master Mentor

@Anshul Sisodia check your networking/firewall on this node.

avatar
Master Mentor

avatar
Expert Contributor

@Artem Ervits no networking or firewall issue as such

avatar

@Anshul Sisodia - It looks like you have a connection issue.

a) Check on the destination host if the datanode is up and running

b) you can run tcpdump between the two hosts and port and monitor the traffic

https://danielmiessler.com/study/tcpdump/

tcpdump is an excellent tool that can give you lot of network related problem information.

avatar
Expert Contributor

a) Datanodes are up and running.

b) Will check the tcpdump and send the output

avatar

The stack trace indicates the DataNode was serving a client block read operation. It attempted to write some data to the client on the socket connection, but the write timed out. This likely indicates a client-side problem, not a DataNode problem. I agree with the assessments to check networking. If you know what client application was running from the "remote" address in the stack trace, then it's also helpful to investigate any logs generated by that application.

avatar
Master Mentor

avatar
Master Mentor

@Anshul Sisodia are you still having problems with this? Can you provide your own solution or accept best answer?