Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

DataXceiver error processing READ_BLOCK operation

DataXceiver error processing READ_BLOCK operation

Rising Star

Frequently getting these below error messages on datanode

ERROR datanode.DataNode (DataXceiver.java:run(250)) - X.X.X.X6:50010:DataXceiver error processing READ_BLOCK operation src: /x.x.x.7:49636 dst: /x.x.x.6:50010 java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/x.x.x.6:50010 remote=/x.x.x.7:49636] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:172) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:220) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:547) at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:716) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:506) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:110) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:232) at java.lang.Thread.run(Thread.java:745)

8 REPLIES 8

Re: DataXceiver error processing READ_BLOCK operation

Mentor

@Anshul Sisodia check your networking/firewall on this node.

Re: DataXceiver error processing READ_BLOCK operation

Mentor

Re: DataXceiver error processing READ_BLOCK operation

Rising Star

@Artem Ervits no networking or firewall issue as such

Re: DataXceiver error processing READ_BLOCK operation

@Anshul Sisodia - It looks like you have a connection issue.

a) Check on the destination host if the datanode is up and running

b) you can run tcpdump between the two hosts and port and monitor the traffic

https://danielmiessler.com/study/tcpdump/

tcpdump is an excellent tool that can give you lot of network related problem information.

Re: DataXceiver error processing READ_BLOCK operation

Rising Star

a) Datanodes are up and running.

b) Will check the tcpdump and send the output

Re: DataXceiver error processing READ_BLOCK operation

The stack trace indicates the DataNode was serving a client block read operation. It attempted to write some data to the client on the socket connection, but the write timed out. This likely indicates a client-side problem, not a DataNode problem. I agree with the assessments to check networking. If you know what client application was running from the "remote" address in the stack trace, then it's also helpful to investigate any logs generated by that application.

Re: DataXceiver error processing READ_BLOCK operation

Re: DataXceiver error processing READ_BLOCK operation

Mentor

@Anshul Sisodia are you still having problems with this? Can you provide your own solution or accept best answer?