I had a datanode failure due to the JAVA HEAP SIZE which caused a huge number of under replicated blocks since there were writes that happened when the node failed.I fixed the JAVA HEAP size and got the node alive.When I'm trying to re-replicate the blocks as mentioned here the number doesn't seem to comedown even when the setrep operation running.
Also, the other thing I observed was the data looks skewed on the datanodes:
Is the skewed data interfering with the setrep operation?
IS there a way I can deal with the skew and the under replicated blocks?
@Rahul Soni I tried to run the balancer as
hdfs balancer -source <overloadedhost>
It ran in 3iterations saying it needs to transfer around 100gb and ended. There were no errors. But it didn’t fix the imbalance.