issue HDFS Disk balancing

[#1]

we are running a 4 DataNodes HDFS cluster. below is the disk details of each DataNodes.

Node1:

Filesystem Size Used Avail Use% Mounted on

/dev/sdc1 2.2T 1.5T 571G 73% /hadoop

/dev/mapper/rootvg-root_lv 4.0G 3.6G 418M 90% /

Node2:

Filesystem Size Used Avail Use% Mounted on

/dev/sdc 1.1T 254G 791G 25% /hadoop

/dev/sdb1 734G 203G 494G 30% /hadoop1

Node3:

Filesystem Size Used Avail Use% Mounted on

/dev/sdb 1.1T 1015G 30G 98% /hadoop

/dev/mapper/rootvg-root_lv 4.0G 3.6G 486M 89%

Node4:

Filesystem Size Used Avail Use% Mounted on

/dev/sdc 1.1T 246G 799G 24% /hadoop

/dev/sdb1 734G 200G 497G 29% /hadoop1

we tried HDFS balancer from Ambari-UI as well as from CLI. but we are not able to evenly rebalance the HDFS DataNodes disks.

below is the cli command which we used.

hdfs balancer -Ddfs.balancer.movedWinWidth=5400000 -Ddfs.balancer.moverThreads=1000 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.datanode.balance.bandwidthPerSec=100000000 -Ddfs.balancer.max-size-to-move=10737418240 -threshold 5

HDFS version is 2.7.1.2.3

any suggestion or advice how we can achieve the rebalancing would be much appreciated.

[#2]

Did you check the namenode log after execute cli?

[#3]

Peter,

we did not received any error logs in name node logs files.

[#4]

I mean, if you are executing cli, then balancing process log is recorded into the namenode log file. Such as INFO messages of balancer.