Reading zero bytes issue in apache hadoop

HadoopCommunity — Thu, 29 Aug 2024 05:17:12 GMT

24/08/29 10:21:15 WARN DataStreamer: Exception for BP-942923949-172.20.0.202-1722967626672:blk_1074290600_560526
Aug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: java.io.EOFException: Unexpected EOF while trying to read response from server
Aug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:521)
Aug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:213)
Aug 29 10:21:15 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.DataStreamer$ResponseProcessor.run(DataStreamer.java:1137)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: 24/08/29 10:27:30 WARN DataStreamer: DataStreamer Exception
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: java.net.SocketTimeoutException: 495000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/172.20.0.194:40396 remote=/172.20.0.194:9866]
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:163)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:158)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:116)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at java.io.DataOutputStream.write(DataOutputStream.java:107)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.DFSPacket.writeTo(DFSPacket.java:193)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.DataStreamer.sendPacket(DataStreamer.java:857)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:762)
Aug 29 10:27:30 gws-siem-dnt-3 bash[2218954]: 24/08/29 10:27:30 WARN DataStreamer: Error Recovery for BP-942923949-172.20.0.202-1722967626672:blk_1074290600_560526 in pipeline [DatanodeInfoWithStorage[172.20.0.194:9866,DS-d8020c9c-24d1-462a-b707-0d219ce5848a,DISK], DatanodeInfoWithStorage[172.20.0.204:9866,DS-5a287a01-fda4-4ad0-a3e6-9d24cc6d8568,DISK], DatanodeInfoWithStorage[172.20.0.196:9866,DS-90f354a6-9531-49dd-97c4-af388f697c34,DISK]]: datanode 0(DatanodeInfoWithStorage[172.20.0.194:9866,DS-d8020c9c-24d1-462a-b707-0d219ce5848a,DISK]) is bad.
last 2 months we have had the issue please help me due to this our script got stuck and the lag has increased
Please help me

Re: Reading zero bytes issue in apache hadoop

shubham_sharma — Sat, 31 Aug 2024 15:25:47 GMT

Hi @HadoopCommunity

This job is failing due to issues retrieving the block from datanodes. Please check the datanode logs during the issue timestamp to check the actual cause of the issue.

Re: Reading zero bytes issue in apache hadoop

shubham_sharma — Thu, 05 Sep 2024 22:04:19 GMT

Hi @HadoopCommunity

Was your question answered? Please take some time to click on "Accept as Solution" -- If you find a reply useful, say thanks by clicking on the thumbs up button below this post.

question Reading zero bytes issue in apache hadoop in Support Questions

Reading zero bytes issue in apache hadoop

Re: Reading zero bytes issue in apache hadoop

Re: Reading zero bytes issue in apache hadoop