@Jordan Moore what are the risks if we still use only 3 zookeeper servers with 17 kafka machines ? ... View more

so just to summary what you said do you mean that we need min 5 zookeeper server for 17 kafka machines ? or in other words how many zookeeper you suggest for the following cluster nodes: master machines - 3 kafka machines - 17 worker machines - 160 ... View more

@Geoffrey , any suggestion how to ciontinue from this point ? ... View more

@Geoffrey yes I already removed it , and the error is exactly the same error as we already seen , ... View more

@Jeoffrey unfortunately after setting the variables we still have the issue, we restart the HDFS service and also the worker machine but we still see the - "Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try" ... View more

should I set this variable to true ? ( dfs.client.block.write.replace-datanode-on-failure.policy=true ) , in Custom hdfs-site ... View more

I removed this variable from the Custom hdfs-site, ... View more

@Geoffrey I set the following in Custom hdfs-site , per your recommendation , and we restart the HDFS service dfs.client.block.write.replace-datanode-on-failure.best-effort=true dfs.client.block.write.replace-datanode-on-failure.enable=false but still the errors still defined from the log ( on the first Datanode machine - worker01 ) ---2018-07-15T08:14:49.049 ERROR [driver][][] [org.apache.spark.scheduler.LiveListenerBus] Listener EventLoggingListener threw an exception java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage:50010,DS-f5c5260a-20b1-43f4-b8fd-53e88db2e48e,DISK], DatanodeInfoWithStorage[:50010,DS-b4758979-52a2-4238-99f0-1b5ec45a7e25,DISK]], original=[DatanodeInfoWithStorage[:50010,DS-f5c5260a-20b1-43f4-b8fd-53e88db2e48e,DISK], DatanodeInfoWithStorage[:50010,DS-b4758979-52a2-4238-99f0-1b5ec45a7e25,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1059) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1122) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1280) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1005) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:512) ... View more