New hdfs balancer the command (tried via CM as well same result) "hdfs balancer" command is slow to move data around to balance the nodes. I have tried it with different thresholds and poicies. The old hdfs balancer in version cdh4.1.3 is pretty fast. It actually moves around data at the speed specified by dfs.balance.bandwidthPerSec. The old cluster moves data around rapidly, ie the balancer thread doesnt sleep. The new one however, seems to either ignore it or is superceeded by another value that is not tunable. The balancer sleeps for 35s then moves a few blocks around, then sleeps again. I have dfs.balance.bandwidthPerSec set to 1GB/s (or 10Gb) Yet the cluster only gets balanced at 15MB/s which is really slow when you have TBs of data. This cluster is completely idle of 18 nodes, nothing is happening other than prep work to move data around and testing to get the cluster to prod status. If anyone knows the quirks of the new hadoop please help! At this snail's pace adding / remove nodes would be disasterous to the hadoop infrastructure. This would indeed become small data platform not big data.
... View more
