How can I use balancer option "setBalancerBandwidth" with cloudera manager ?

[#1]

Hi, all.

I use CDH 5.7 and cloudera manager for cluster operaitions.

I'd like to use balancer option "setBalancerBandwidth" for HDFS balancing, but I read admin_hdfs_balancer.html and found "If you use Cloudera Manager, do not use these command-line instructions".

How can I use balancer option "setBalancerBandwidth" with cloudera manager ?

Or, above comments("If you use...") is not valid for "dfsadmin -setBalancerBandwidth" ?

I know that "dfs.datanode.balance.bandwidthPerSec" option is set bandwith for hdfs balancer, but I would rather not use this option because of need to reboot namenodes.

[#2 · Accepted solution]

Hi,

It you set the balancer bandwidth in Cloudera manager, then when the datanodes are started, they will have that bandwidth seting for balancing operations. However, using the command line it is possible to change the bandwidth while the datanodes or balancer is running, without restarting them. If you do this, you just need to remember that if you restart any datanodes, the bandwidth setting will revert back to that set in Cloudera Manager and you will need to run the command again.

To set the balancer bandwidth from the command line and without a restart you can run the following:

sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 104857600

If you have HA setup for HDFS, the above command may fail, and you should check which is the active namenode and run the command like follows (substituting the correct hostname for activeNamenode below):

sudo -u hdfs hdfs dfsadmin -fs hdfs://activeNamenode:8020/ -setBalancerBandwidth 104857600

To check this command worked, the following log entries should appear within the Datanode log files:

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_BALANCERBANDWIDTHUPDATE
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Updating balance throttler bandwidth from 10485760 bytes/s to: 104857600 bytes/s.