Support Questions
Find answers, ask questions, and share your expertise

Data Node Exception when writing file To Hadoop

Data Node Exception when writing file To Hadoop

iam using hadoop apache 2.7.1 and it's high available
and when issuing the command

hdfs dfs -put file1 /hadoophome/

iam not able to put my file with following log in one of the available data nodes


i have 6 data nodes and the replication factor is 3

2017-08-22 15:01:07,351 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111953_371166, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:07,938 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1237)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:08,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111954_371167 src: /192.168.25.8:35957 dest: /192.168.25.2:50010
2017-08-22 15:01:09,146 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.25.8:35957, dest: /192.168.25.2:50010, bytes: 82, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1028946205_23, offset: 0, srvID: 075f3a14-9b13-404d-8ba8-4066212655d7, blockid: BP-840293587-192.168.25.22-1499689510217:blk_1074111954_371167, duration: 3404900
2017-08-22 15:01:09,146 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111954_371167, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:09,280 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.run():
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48085]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1473)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1410)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1323)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:09,281 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162, type=HAS_DOWNSTREAM_IN_PIPELINE
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48085]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1473)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1410)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1323)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:09,281 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:10,808 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:3173ms (threshold=300ms)
2017-08-22 15:01:10,808 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:10,809 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162 received exception java.nio.channels.ClosedByInterruptException
2017-08-22 15:01:10,809 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: dn2:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.25.2:48085 dst: /192.168.25.2:50010
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,314 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162 src: /192.168.25.2:48091 dest: /192.168.25.2:50010
2017-08-22 15:01:11,314 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover RBW replica BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162
2017-08-22 15:01:11,314 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_1074111949_371162, RBW
getNumBytes() = 53932032
getBytesOnDisk() = 53932032
getVisibleLength()= 53932032
getVolume() = /hdd/data_dir/current
getBlockFile() = /hdd/data_dir/current/BP-840293587-192.168.25.22-1499689510217/current/rbw/blk_1074111949
bytesAcked=53932032
bytesOnDisk=53932032
2017-08-22 15:01:11,619 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1237)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48091]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169, type=HAS_DOWNSTREAM_IN_PIPELINE: Thread is interrupted.
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169 received exception java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48091]
2017-08-22 15:01:11,631 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: dn2:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.25.2:48091 dst: /192.168.25.2:50010
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48091]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,959 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111956_371170 src: /192.168.25.8:35958 dest: /192.168.25.2:50010
2017-08-22 15:01:11,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.25.8:35958, dest: /192.168.25.2:50010, bytes: 82, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_899288339_23, offset: 0, srvID: 075f3a14-9b13-404d-8ba8-4066212655d7, blockid: BP-840293587-192.168.25.22-1499689510217:blk_1074111956_371170, duration: 22005000
2017-08-22 15:01:11,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111956_371170, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2017-08-22 15:01:15,027 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111957_371172 src: /192.168.25.2:48097 dest: /192.168.25.2:50010
2017-08-22 15:01:15,037 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.25.2:48097, dest: /192.168.25.2:50010, bytes: 82, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1375829046_24, offset: 0, srvID: 075f3a14-9b13-404d-8ba8-4066212655d7, blockid: BP-840293587-192.168.25.22-1499689510217:blk_1074111957_371172, duration: 3663100
2017-08-22 15:01:15,038 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111957_371172, type=HAS_DOWNSTREAM_IN_PIPELINE terminating

any help to determine what is the problem please?

1 REPLY 1
Highlighted

Re: Data Node Exception when writing file To Hadoop

Explorer

Hi @oula.alshiekh@gmail.com alshiekh ,

From the above stacktraces it looks like, socket timeouts are set to very less values as 300ms.
Hadoop's default values are ReadTimeout=60000, WriteTimeout=8*60000

Please check below configurations in Datanode's configs,
"dfs.client.socket-timeout"
"dfs.datanode.socket.write.timeout"

If the values are set to 300ms, then please increase these values and restart the datanodes.

Don't have an account?