Support Questions
Find answers, ask questions, and share your expertise

hdfs balancer

New Contributor
Hello,
  • I started my balancer job as below:

nohup hdfs balancer -threshold 40 -include hostfile >rebal.log 2>rebal.err &

  • The rebal.err is showing files being moved:

16/09/07 13:47:42 INFO net.NetworkTopology: Adding a new node: /default-rack/10.12.217.87:50010

16/09/07 13:47:42 INFO net.NetworkTopology: Adding a new node: /default-rack/10.12.217.96:50010

16/09/07 13:47:42 INFO net.NetworkTopology: Adding a new node: /default-rack/10.12.217.94:50010

16/09/07 13:47:42 INFO net.NetworkTopology: Adding a new node: /default-rack/10.12.217.95:50010

16/09/07 13:47:42 INFO balancer.Balancer: 1 over-utilized: [Source[10.12.217.87:50010, utilization=94.3875712244642]] 16/09/07 13:47:42 INFO balancer.Balancer: 0 underutilized: [] 16/09/07 13:47:42 INFO balancer.Balancer: Need to move 2.08 TB to make the cluster balanced. 16/09/07 13:47:42 INFO balancer.Balancer: Decided to move 10 GB bytes from 10.12.217.87:50010 to 10.12.217.96:50010 16/09/07 13:47:42 INFO balancer.Balancer: Will move 10 GB in this iteration 16/09/07 13:47:43 INFO balancer.Balancer: Successfully moved blk_1121226541_47486631 with size=21089 from 10.12.217.87:50010 to 10.12.217.96:5001 0 through 10.12.217.95:50010

  • but the rebal.log shows 'Bytes Already Moved' remaining at 0B after 46 iterations

Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved

07-Sep-2016 13:44:36 0 0 B 2.08 TB 10 GB

...

07-Sep-2016 14:13:05 46 0 B 2.08 TB 10 GB

  • It has been running for hours, but i don't see changes in 'df' on the over-utilized hosts. It looks like the rebalancer is not actually moving blocks. Is there any way i can see more verbose output or am i missing something?

Thanks in advance for any help.

2 REPLIES 2

Re: hdfs balancer

Contributor

@khanh vo trying checking namenode logs. It provides more information of block assignment.

Re: hdfs balancer

New Contributor

@Vinod Bonthu checking namenode log files: hadoop-hdfs-journalnode, gc.log, hadoop-hdfs-namenode, hdfs-audit.log but I don't see any references to the balancer job.