hi all we have ambari cluster with 4 datanode machine ( workers ) , and on each worker machine we have 1 disk of 1T size before I explain the problem I want to clear that we verify the following and we not see any problem on the following subject 1 cluster is working without network problem 2 we check the DNS and resolving hostname is correctly 3 java heap size on HDFS increase to 8G ( so no problem with java heap size ) 5. we checked the HDFS service check and no issue with that 6. we set the following: To resolve this issue, we set the following two properties from Ambari > HDFS > Configs > Custom HDFS site > Add Property: dfs.client.block.write.replace-datanode-on-failure.enable=NEVER dfs.client.block.write.replace-datanode-on-failure.policy=NEVER but we still have the problem NOW - lets talk about the problem: on one of the worker machine we see that tail -f /grid/sdb/hadoop/yarn/log/application_1523836627832749_4432/container_e23_1592736529519_4432_01_000041/stderr ---2018-07-12T20:51:28.028 ERROR [driver][][] [org.apache.spark.scheduler.LiveListenerBus] Listener EventLoggingListener threw an exception java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available to try. (Nodes: current=[DatanodeInfoWithStorage[45.23.2.56:50010,DS-f5c5260a-20b1-43f4-b8fd-53e88db2e48e,DISK], DatanodeInfoWithStorage[45.23.2.56:50010,DS-b4758979-52a2-4238-99f0-1b5ec45a7e25,DISK]], original=[DatanodeInfoWithStorage[45.23.2.56:50010,DS-f5c5260a-20b1-43f4-b8fd-53e88db2e48e,DISK], DatanodeInfoWithStorage[45.23.2.56:50010,DS-b4758979-52a2-4238-99f0-1b5ec45a7e25,DISK]]). The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration. at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.findNewDatanode(DFSOutputStream.java:1059) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1122) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1280) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:1005) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:512) we can saw the error about - java.io.IOException: Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available what we can do else in order to resolved the failed "Failed to replace a bad datanode on the existing pipeline due to no more good datanodes being available" ? ... View more

hi all according to Hortonworks recommendation ( 2. File System Partitioning Recommendations Setting Up File System Partitions Use the following as a base configuration for all nodes in your cluster: • Root partition: OS and core program files • Swap: Size 2X system memory ) we can see that Hortonworks suggest that swap on machine should be twice from the memory for example if we set 32G on masters machines , then swap should be 64G but actually , from system side this could be risk the performance because swap is twice then memory lets give here example when memory resource are ended , then OS use the swap swap is very slow memory , so ambari cluster will be negative affected when major swap resource will be in used so from pure system point is very hard to accept that swap should be twice then memory I will happy to get remarks , or ideas according to swap , and what are the real value of swap against memory if we not want to risk the cluster performance from Hortonworks doc: 2. File System Partitioning Recommendations Setting Up File System Partitions Use the following as a base configuration for all nodes in your cluster: • Root partition: OS and core program files • Swap: Size 2X system memory ... View more

hi all we have issue on data-node machine ( worker machine ) we notice that /var became full and that because stdout file is 89G example: /var/Hadoop/yarn/log/application 183625335_0110/container_e54_1532846235_0180_02_025216/stdout please advise what could be the reason for this issue and how to solve it so stdout will not became as huge size ... View more