Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

issue HDFS Disk balancing

issue HDFS Disk balancing

New Contributor

we are running a 4 DataNodes HDFS cluster. below is the disk details of each DataNodes.

Node1:

Filesystem Size Used Avail Use% Mounted on

/dev/sdc1 2.2T 1.5T 571G 73% /hadoop

/dev/mapper/rootvg-root_lv 4.0G 3.6G 418M 90% /

Node2:

Filesystem Size Used Avail Use% Mounted on

/dev/sdc 1.1T 254G 791G 25% /hadoop

/dev/sdb1 734G 203G 494G 30% /hadoop1

Node3:

Filesystem Size Used Avail Use% Mounted on

/dev/sdb 1.1T 1015G 30G 98% /hadoop

/dev/mapper/rootvg-root_lv 4.0G 3.6G 486M 89%

Node4:

Filesystem Size Used Avail Use% Mounted on

/dev/sdc 1.1T 246G 799G 24% /hadoop

/dev/sdb1 734G 200G 497G 29% /hadoop1

we tried HDFS balancer from Ambari-UI as well as from CLI. but we are not able to evenly rebalance the HDFS DataNodes disks.

below is the cli command which we used.

hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5

HDFS version is 2.7.1.2.3


any suggestion or advice how we can achieve the rebalancing would be much appreciated.

3 REPLIES 3

Re: issue HDFS Disk balancing

New Contributor

Did you check the namenode log after execute cli?

Re: issue HDFS Disk balancing

New Contributor

Peter,

we did not received any error logs in name node logs files.

Highlighted

Re: issue HDFS Disk balancing

New Contributor

I mean, if you are executing cli, then balancing process log is recorded into the namenode log file. Such as INFO messages of balancer.

Don't have an account?
Coming from Hortonworks? Activate your account here