Data Node Exception when writing file To Hadoop

[#1]

iam using hadoop apache 2.7.1 and it's high available
and when issuing the command

hdfs dfs -put file1 /hadoophome/

iam not able to put my file with following log in one of the available data nodes


i have 6 data nodes and the replication factor is 3

2017-08-22 15:01:07,351 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111953_371166, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:07,938 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1237)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:08,984 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111954_371167 src: /192.168.25.8:35957 dest: /192.168.25.2:50010
2017-08-22 15:01:09,146 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.25.8:35957, dest: /192.168.25.2:50010, bytes: 82, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1028946205_23, offset: 0, srvID: 075f3a14-9b13-404d-8ba8-4066212655d7, blockid: BP-840293587-192.168.25.22-1499689510217:blk_1074111954_371167, duration: 3404900
2017-08-22 15:01:09,146 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111954_371167, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:09,280 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in BlockReceiver.run():
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48085]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1473)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1410)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1323)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:09,281 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162, type=HAS_DOWNSTREAM_IN_PIPELINE
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48085]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1473)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstream(BlockReceiver.java:1410)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1323)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:09,281 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:10,808 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Slow BlockReceiver write data to disk cost:3173ms (threshold=300ms)
2017-08-22 15:01:10,808 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:10,809 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162 received exception java.nio.channels.ClosedByInterruptException
2017-08-22 15:01:10,809 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: dn2:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.25.2:48085 dst: /192.168.25.2:50010
java.nio.channels.ClosedByInterruptException
at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:417)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,314 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162 src: /192.168.25.2:48091 dest: /192.168.25.2:50010
2017-08-22 15:01:11,314 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recover RBW replica BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371162
2017-08-22 15:01:11,314 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Recovering ReplicaBeingWritten, blk_1074111949_371162, RBW
getNumBytes() = 53932032
getBytesOnDisk() = 53932032
getVisibleLength()= 53932032
getVolume() = /hdd/data_dir/current
getBlockFile() = /hdd/data_dir/current/BP-840293587-192.168.25.22-1499689510217/current/rbw/blk_1074111949
bytesAcked=53932032
bytesOnDisk=53932032
2017-08-22 15:01:11,619 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169, type=HAS_DOWNSTREAM_IN_PIPELINE
java.io.EOFException: Premature EOF: no length prefix available
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280)
at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:244)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:1237)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Exception for BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48091]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169, type=HAS_DOWNSTREAM_IN_PIPELINE: Thread is interrupted.
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2017-08-22 15:01:11,630 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: opWriteBlock BP-840293587-192.168.25.22-1499689510217:blk_1074111949_371169 received exception java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48091]
2017-08-22 15:01:11,631 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: dn2:50010:DataXceiver error processing WRITE_BLOCK operation src: /192.168.25.2:48091 dst: /192.168.25.2:50010
java.net.SocketTimeoutException: 300 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.25.2:50010 remote=/192.168.25.2:48091]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:472)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:849)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:804)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:137)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:74)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:251)
at java.lang.Thread.run(Thread.java:748)
2017-08-22 15:01:11,959 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111956_371170 src: /192.168.25.8:35958 dest: /192.168.25.2:50010
2017-08-22 15:01:11,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.25.8:35958, dest: /192.168.25.2:50010, bytes: 82, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_899288339_23, offset: 0, srvID: 075f3a14-9b13-404d-8ba8-4066212655d7, blockid: BP-840293587-192.168.25.22-1499689510217:blk_1074111956_371170, duration: 22005000
2017-08-22 15:01:11,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111956_371170, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2017-08-22 15:01:15,027 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-840293587-192.168.25.22-1499689510217:blk_1074111957_371172 src: /192.168.25.2:48097 dest: /192.168.25.2:50010
2017-08-22 15:01:15,037 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /192.168.25.2:48097, dest: /192.168.25.2:50010, bytes: 82, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1375829046_24, offset: 0, srvID: 075f3a14-9b13-404d-8ba8-4066212655d7, blockid: BP-840293587-192.168.25.22-1499689510217:blk_1074111957_371172, duration: 3663100
2017-08-22 15:01:15,038 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-840293587-192.168.25.22-1499689510217:blk_1074111957_371172, type=HAS_DOWNSTREAM_IN_PIPELINE terminating

any help to determine what is the problem please?

[#2]

Hi @oula.alshiekh@gmail.com alshiekh ,

From the above stacktraces it looks like, socket timeouts are set to very less values as 300ms.
Hadoop's default values are ReadTimeout=60000, WriteTimeout=8*60000

Please check below configurations in Datanode's configs,
"dfs.client.socket-timeout"
"dfs.datanode.socket.write.timeout"

If the values are set to 300ms, then please increase these values and restart the datanodes.