Created 03-21-2018 06:21 PM
I had a datanode failure due to the JAVA HEAP SIZE which caused a huge number of under replicated blocks since there were writes that happened when the node failed.I fixed the JAVA HEAP size and got the node alive.When I'm trying to re-replicate the blocks as mentioned here the number doesn't seem to comedown even when the setrep operation running.
Also, the other thing I observed was the data looks skewed on the datanodes:
Is the skewed data interfering with the setrep operation?
IS there a way I can deal with the skew and the under replicated blocks?
Created 03-22-2018 07:40 PM
@Swaapnika Guntaka Your problem is exactly why Balancer was created for :)
Follows this and this link to fix your problem.
Let know if that works for you.
Created 03-23-2018 12:50 AM
@Rahul Soni I tried to run the balancer as
hdfs balancer -source <overloadedhost>
It ran in 3iterations saying it needs to transfer around 100gb and ended. There were no errors. But it didn’t fix the imbalance.
Created 03-23-2018 12:58 AM
What HDP version you are using?