Configured Azure WASB storage as a default HDFS location through Cloudbreak, which had made Hadoop local HDFS capacity as 0 in Ambari (100 % utilized). I have default replication as 1 but now when I am trying to decommission a node, datanode tries to rebalance some 28KB of data to another available datanode. However, our HDFS has 0 capacity and therefore, decommissioning fails with below given error: New node(s) could not be removed from the cluster. Reason Trying to move '28672' bytes worth of data to nodes with '0' bytes of capacity is not allowed Getting the information on cluster shows that default local HDFS is still used for some KB space which is getting rebalanced whereas available capacity is 0: "CapacityRemaining" : 0,
"CapacityTotal" : 0,
"CapacityUsed" : 131072,
"DeadNodes" : "{}",
"DecomNodes" : "{}",
"HeapMemoryMax" : 1060372480,
"HeapMemoryUsed" : 147668152,
"NonDfsUsedSpace" : 0,
"NonHeapMemoryMax" : -1,
"NonHeapMemoryUsed" : 75319744,
"PercentRemaining" : 0.0,
"PercentUsed" : 100.0,
"Safemode" : "",
"StartTime" : 1518241019502,
"TotalFiles" : 1,
"UpgradeFinalized" : true,
There is an ambari decommissioned jar used by Cloudbreak to check, if HDFS is running out of space. Is there a way to change this jar? if (remainingSpace < safetyUsedSpace) {
throw new BadRequestException(
String.format("Trying to move '%s' bytes worth of data to nodes with '%s' bytes of capacity is not allowed", usedSpace, remainingSpace)
);
} Reference link: https://github.com/hortonworks/cloudbreak/blob/1.16.4/core/src/main/java/com/sequenceiq/cloudbreak/service/cluster/flow/AmbariDecommissioner.java#L314-L321
... View more
