question Re: hdfs balancer slow to move data around in cdh 5 in Archives of Support Questions (Read Only)

hdfs balancer slow to move data around in cdh 5

kal.pan — Fri, 16 Sep 2022 09:05:31 GMT

New hdfs balancer the command (tried via CM as well same result)

"hdfs balancer" command is slow to move data around to balance the nodes.

I have tried it with different thresholds and poicies.

The old hdfs balancer in version cdh4.1.3 is pretty fast. It actually moves around data at the speed specified by

dfs.balance.bandwidthPerSec. The old cluster moves data around rapidly, ie the balancer thread doesnt sleep.

The new one however, seems to either ignore it or is superceeded by another value that is not tunable. The balancer sleeps for 35s then moves a few blocks around, then sleeps again.

I have dfs.balance.bandwidthPerSec set to 1GB/s (or 10Gb)

Yet the cluster only gets balanced at 15MB/s which is really slow when you have TBs of data.

This cluster is completely idle of 18 nodes, nothing is happening other than prep work to move data around and testing to get the cluster to prod status.

If anyone knows the quirks of the new hadoop please help! At this snail's pace adding / remove nodes would be disasterous to the hadoop infrastructure. This would indeed become small data platform not big data.

Re: hdfs balancer slow to move data around in cdh 5

Rolando — Tue, 19 Aug 2014 22:07:08 GMT

More than likely you are hitting HDFS-6621. This is fairly new so it hasn't been fixed in CDH5 as of yet.

Re: hdfs balancer slow to move data around in cdh 5

kal.pan — Tue, 19 Aug 2014 22:47:01 GMT

The issue sounds close, but issue I am seeing that the thread sleeps for 35s and moves more data.

It doesnt exit out. It continues to run, but sleeps 35s then moves, then sleeps, then moves.

I gathered total rate to be about 15MB/s doing this it this way.

like so:

14/08/14 14:22:44 INFO balancer.Balancer: Successfully moved blk_1075451687_1713835 with size=132482663 from 10.2.2.3:50010 to 10.2.1.247:50010 through 10.2.1.248:50010
14/08/14 14:23:19 INFO net.NetworkTopology: Adding a new node: /default/10.2.1.253:50010

14/08/14 14:23:22 INFO balancer.Balancer: Successfully moved blk_1076792220_3054863 with size=134217728 from 10.2.1.253:50010 to 10.2.1.247:50010 through 10.2.2.7:50010
14/08/14 14:23:56 INFO net.NetworkTopology: Adding a new node: /default/10.2.2.2:50010

You can see the last move and next time it wakesup todo the move again its about 35s every time!

Re: hdfs balancer slow to move data around in cdh 5

GautamG — Thu, 23 Oct 2014 04:11:07 GMT

Sorry for the late response. What exact version of CDH 5.x are you running?

Re: hdfs balancer slow to move data around in cdh 5

kal.pan — Thu, 23 Oct 2014 18:06:49 GMT

There was a patch released that fixed this problem.

I updated the bundle give to me by support.

Re: hdfs balancer slow to move data around in cdh 5

GautamG — Fri, 24 Oct 2014 02:34:46 GMT

That's great news, thanks for the feedback

Re: hdfs balancer slow to move data around in cdh 5

eliran007 — Wed, 24 Dec 2014 09:58:32 GMT

it seems i have the same issue exactly with CDH 5.1.0

where can i get the patch you are talking about?

Thanks,

Eliran

Re: hdfs balancer slow to move data around in cdh 5

GautamG — Wed, 24 Dec 2014 11:17:58 GMT

The patch is part of CDH 5.1.4 and CDH 5.2.0 (and later versions). You can simply upgrade to v5.1.4 from v5.1.0 for the fix.

Re: hdfs balancer slow to move data around in cdh 5

eliran007 — Wed, 24 Dec 2014 15:41:28 GMT

god bless you!!!

i have been rebalancing fro almost two month and its been on 40%, and now in and hour it finished almost 10%

we were going crazy here.

Thanks alot

Re: hdfs balancer slow to move data around in cdh 5

jakeri — Fri, 16 Jan 2015 14:56:07 GMT

Hello,

We are also experiencing this. Could you point us to which patch this is?

Kind regards,

Jakob

Re: hdfs balancer slow to move data around in cdh 5

kal.pan — Fri, 16 Jan 2015 20:16:47 GMT

We have created PATCH-434 for your s8cluster that is a backport of HDFS-6621 for CDH 5.1.0.
This patch was tested against CentOS 5.7 and CentOS 6+ which are our supported OS versions for CDH 5.1.0.

Re: hdfs balancer slow to move data around in cdh 5

GautamG — Sat, 17 Jan 2015 23:49:56 GMT

@jakeri wrote:
Hello,

We are also experiencing this. Could you point us to which patch this is?

Kind regards,
Jakob

The patch is part of CDH 5.1.4 and CDH 5.2.0 (and later versions). You can simply upgrade to v5.1.4 from v5.1.0 for the fix. If you have a support contract, you can log a case for a patch on a specific version of CDH.