Support Questions

iamfromsky · ‎08-02-2021

i have found my one of CDH has so many errors on every datanode, the error logs as below. who have this kind experience on this issue ? and give me some advises

2021-08-03 05:23:43,389 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061604_2849065475 src: /10.37.54.218:36088 dest: /10.37.54.218:1004
2021-08-03 05:23:43,700 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.218:36082, dest: /10.37.54.218:1004, bytes: 358, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-859199005_222, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061597_2849065468, duration: 59733778
2021-08-03 05:23:43,700 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061597_2849065468, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-08-03 05:23:43,833 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.218:36088, dest: /10.37.54.218:1004, bytes: 309, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-859199005_222, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061604_2849065475, duration: 200220559
2021-08-03 05:23:43,833 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061604_2849065475, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-08-03 05:23:44,044 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061619_2849065490 src: /10.37.54.15:59320 dest: /10.37.54.218:1004
2021-08-03 05:23:44,058 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.15:59320, dest: /10.37.54.218:1004, bytes: 112, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1165227557_139, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061619_2849065490, duration: 3752037
2021-08-03 05:23:44,058 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061619_2849065490, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-08-03 05:23:45,037 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061679_2849065550 src: /10.37.54.218:36108 dest: /10.37.54.218:1004
2021-08-03 05:23:45,185 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.218:36108, dest: /10.37.54.218:1004, bytes: 1415899, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1849481388_3452, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061679_2849065550, duration: 61038196
2021-08-03 05:23:45,185 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061679_2849065550, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-08-03 05:23:45,497 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Moved BP-2123011416-10.37.54.12-1457006347704:blk_3802213701_2741214333 from /10.37.54.13:44312, delHint=6a0ea409-35ad-42c5-956d-44a5b9bd58a6
2021-08-03 05:23:45,703 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061646_2849065517 src: /10.37.54.216:54728 dest: /10.37.54.218:1004
2021-08-03 05:23:45,714 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-2123011416-10.37.54.12-1457006347704:blk_3910061646_2849065517 src: /10.37.54.216:54728 dest: /10.37.54.218:1004 of size 4786053
2021-08-03 05:23:45,998 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Moved BP-2123011416-10.37.54.12-1457006347704:blk_1842563008_775812314 from /10.37.54.13:50434, delHint=6a0ea409-35ad-42c5-956d-44a5b9bd58a6
2021-08-03 05:23:46,042 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:605)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:789)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:736)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:551)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
        at java.lang.Thread.run(Thread.java:745)
2021-08-03 05:23:46,043 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: BlockSender.sendChunks() exception: 
java.io.IOException: 断开的管道
        at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
        at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428)
        at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493)
        at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:223)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendPacket(BlockSender.java:605)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.doSendBlock(BlockSender.java:789)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:736)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:551)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:246)
        at java.lang.Thread.run(Thread.java:745)
2021-08-03 05:23:47,003 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061723_2849065594 src: /10.37.54.216:54770 dest: /10.37.54.218:1004
2021-08-03 05:23:47,018 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061724_2849065595 src: /10.37.54.216:54772 dest: /10.37.54.218:1004
2021-08-03 05:23:47,019 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.216:54772, dest: /10.37.54.218:1004, bytes: 4158, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-1438538333_1, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061724_2849065595, duration: 1392081
2021-08-03 05:23:47,019 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061724_2849065595, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2021-08-03 05:23:47,048 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061725_2849065596 src: /10.37.54.216:54774 dest: /10.37.54.218:1004
2021-08-03 05:23:47,056 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.216:54774, dest: /10.37.54.218:1004, bytes: 69, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1452909160_189, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061725_2849065596, duration: 7712861
2021-08-03 05:23:47,056 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061725_2849065596, type=LAST_IN_PIPELINE, downstreams=0:[] terminating
2021-08-03 05:23:47,371 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061731_2849065602 src: /10.37.54.218:36198 dest: /10.37.54.218:1004
2021-08-03 05:23:47,407 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.218:36198, dest: /10.37.54.218:1004, bytes: 314, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_466653976_222, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061731_2849065602, duration: 11069615
2021-08-03 05:23:47,407 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061731_2849065602, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-08-03 05:23:47,422 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-2123011416-10.37.54.12-1457006347704:blk_3910061732_2849065603 src: /10.37.54.218:36202 dest: /10.37.54.218:1004
2021-08-03 05:23:47,458 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.37.54.218:36202, dest: /10.37.54.218:1004, bytes: 17456, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_466653976_222, offset: 0, srvID: 44713da0-9f69-44ea-b6c0-8f7420a41f83, blockid: BP-2123011416-10.37.54.12-1457006347704:blk_3910061732_2849065603, duration: 9623611
2021-08-03 05:23:47,458 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-2123011416-10.37.54.12-1457006347704:blk_3910061732_2849065603, type=HAS_DOWNSTREAM_IN_PIPELINE terminating
2021-08-03 05:23:47,497 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Moved BP-2123011416-10.37.54.12-1457006347704:blk_2529434549_1466543157 from /10.37.54.13:39396, delHint=6a0ea409-35ad-42c5-956d-44a5b9bd58a6

iamfromsky · ‎08-12-2021

i give you more details about this cdh cluster. the original cluster is 5.14 and os version is Centos 6.5, parcels REHL6, and recently i have added new machines into this cluster, os version is Centos 7.6 parcels is REHL7.

all this erros happend just on the new machines which is REHL 7. the old datanode doesn't have this errors.

View solution in original post

Madhur · ‎08-04-2021

Hi @iamfromsky ,

Thank you for reaching out to our community!

The error message which you have provided is logged when either a "broken pipe" or "connection reset" happens, which is most likely network-related.

Please check if the network is stable when you see these errors.

Also refer Jira HDFS-8814 for more details.

Madhuri Adipudi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Community Guidelines
How to use the forum

iamfromsky · ‎08-12-2021

i give you more details about this cdh cluster. the original cluster is 5.14 and os version is Centos 6.5, parcels REHL6, and recently i have added new machines into this cluster, os version is Centos 7.6 parcels is REHL7.

all this erros happend just on the new machines which is REHL 7. the old datanode doesn't have this errors.

iamfromsky · ‎03-23-2022

oh, this is a long time ago issue, the root cause is because new machines charset is not utf-8, just keep all the machines chaset is utf-8 , then its ok.

Support Questions

BlockSender.sendChuncks() error