https://community.cloudera.com/t5/Support-Questions/DATANODE-Failed-to-replace-a-bad-datanode-on-the-existing/m-p/184142#M146289 <P>hi all</P><P>we have ambari cluster with 4 datanode machine ( workers ) , and on each worker machine we have 1 disk of 1T size</P><P>before I explain the problem I want to clear that we verify the following and we not see any problem on the following subject</P><P><EM><STRONG>1 cluster is working without network problem</STRONG></EM></P><P><EM><STRONG>2 we check the DNS and resolving hostname is correctly</STRONG></EM></P><P><EM><STRONG>3 java heap size on HDFS increase to 8G ( so no problem with java heap size )</STRONG></EM></P><P><EM><STRONG>5. we checked the HDFS service check and no issue with that</STRONG></EM></P><P><EM><STRONG>6. we set the following:</STRONG></EM></P><P><EM><STRONG>To resolve this issue, we set the following two properties from Ambari &gt; HDFS &gt; Configs &gt; Custom HDFS site &gt; Add Property:</STRONG></EM></P><P><EM><STRONG>dfs.client.block.write.replace-datanode-on-failure.enable=NEVER</STRONG></EM></P><P><EM><STRONG>dfs.client.block.write.replace-datanode-on-failure.policy=NEVER</STRONG></EM></P><P><STRONG>but we still have the problem</STRONG></P><P><STRONG><BR /></STRONG></P><P><STRONG><BR /></STRONG></P><P>NOW - lets talk about the problem:</P><P>on one of the worker machine we see that</P><PRE> tail -f /grid/sdb/hadoop/yarn/log/application_1523836627832749_4432/container_e23_1592736529519_4432_01_000041/stderr ---2018-07-12T20:51:28.028 ERROR [driver][][] [org.apache.spark.scheduler.LiveListenerBus] Listener EventLoggingListener threw an exception java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[45.23.2.56:50010,DS-f5c5260a-20b1-43f4-b8fd-53e88db2e48e,DISK], DatanodeInfoWithStorage[45.23.2.56:50010,DS-b4758979-52a2-4238-99f0-1b5ec45a7e25,DISK]], original=[DatanodeInfoWithStorage[45.23.2.56:50010,DS-f5c5260a-20b1-43f4-b8fd-53e88db2e48e,DISK], DatanodeInfoWithStorage[45.23.2.56:50010,DS-b4758979-52a2-4238-99f0-1b5ec45a7e25,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1059) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1122) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1280) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1005) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:512) </PRE><P> we can saw the error about - j<STRONG>ava.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available </STRONG></P><P> what we can do else in order to resolved the failed "<STRONG>Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available</STRONG>" ?</P> Fri, 13 Jul 2018 04:13:36 GMT mike_bronson7 2018-07-13T04:13:36Z