I have got HDP-126.96.36.199 cluster with 8 data nodes. Each node has 256 GB RAM, Disk 12*6 TB, CPU 2*8. I am experiencing extremely slow HDFS write performance, all the data nodes are reported live by ambari but few of them are not sending the block reports to namenode on regular interval.This interval is more than 3 days for some of them.I am also getting below INFO messages in hadoop-hdfs-datanode-.log file for all the data nodes.
2018-05-27 18:07:36,945 INFO datanode.DataNode (BlockReceiver.java:handleMirrorOutError(429)) - DatanodeRegistration(:50010, datanodeUuid=19d031bb-5452-450e-8199-46e7732ab682, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-717d1e37-d2a1-47e8-9623-d0e3069e7134;nsid=1980284380;c=0):Exception writing BP-1033621575-10.217.99.11-1507285615620:blk_1345492915_271786962 to mirror :50010
Please guide to resolve this issue. Thanks in advance for your help.