04-29-2015 11:04 AM
The Balancer isn't running on its own. I have run run it manually from the command line.
Some of my HDFS nodes got to 97% full, whle others were only 30-something% full.
It works fine from the command line.
How can I check to see why it is not running on its own? As I understand it, it is supposed to do so.
04-29-2015 03:34 PM - edited 04-29-2015 03:39 PM
It could actually be running and just not aggressively enough.
I've discovered the median size of our HDFS files is something like 6KB, so the balancer is fairly inefficient, since its execution time is the more or less the same for each block, given a fairly fast network.
I'm going to have to scedule the balancer to run from cron once a day or so.
So my queston is slightly modified: How can I tell when the balancer runs when the Balancer service is configured? There do not seem to be any parameters related to scheduling it.
One related question: How does the balancer choose which blocks to move? Does it favor small files over large ones? The reason is because I used the output of the balancer ("moved block blah with size=..."), which includes the size of each block, as a sample of my file sizes. We actually have a slighty less than 1 to 1 blocks-to-file ratio, and of the 32000 files I sampled form the balancer run, only 2000 or were "full" blocks of 64MB.
04-30-2015 10:57 AM
10-11-2015 10:17 PM
Do you mean to say the role Balancer in hdfs doesnt work balancing the data?
Do we have to run balancer from backend for sure?
is there any other way?
Thanks in advance
01-03-2017 10:25 AM