Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Data Nodes slow performance

Data Nodes slow performance

Contributor

Hi All,

I have got HDP-2.6.0.3 cluster with 8 data nodes. Each node has 256 GB RAM, Disk 12*6 TB, CPU 2*8. I am experiencing extremely slow HDFS write performance, all the data nodes are reported live by ambari but few of them are not sending the block reports to namenode on regular interval.This interval is more than 3 days for some of them.I am also getting below INFO messages in hadoop-hdfs-datanode-.log file for all the data nodes.

2018-05-27 18:07:36,945 INFO datanode.DataNode (BlockReceiver.java:handleMirrorOutError(429)) - DatanodeRegistration(:50010, datanodeUuid=19d031bb-5452-450e-8199-46e7732ab682, infoPort=50075, infoSecurePort=0, ipcPort=8010, storageInfo=lv=-56;cid=CID-717d1e37-d2a1-47e8-9623-d0e3069e7134;nsid=1980284380;c=0):Exception writing BP-1033621575-10.217.99.11-1507285615620:blk_1345492915_271786962 to mirror :50010

Please guide to resolve this issue. Thanks in advance for your help.

hadoop-hdfs-datanode.png namenode-ui.pngdatanode-ui.png