Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I use balancer option "setBalancerBandwidth" with cloudera manager ?

SOLVED Go to solution

How can I use balancer option "setBalancerBandwidth" with cloudera manager ?

New Contributor

Hi, all.

I use CDH 5.7 and cloudera manager for cluster operaitions.

 

I'd like to use balancer option "setBalancerBandwidth" for HDFS balancing, but I read admin_hdfs_balancer.html and found "If you use Cloudera Manager, do not use these command-line instructions".

 

How can I use balancer option "setBalancerBandwidth" with cloudera manager ?

 

Or, above comments("If you use...") is not valid for "dfsadmin -setBalancerBandwidth" ?

 

I know that "dfs.datanode.balance.bandwidthPerSec" option is set bandwith for hdfs balancer, but I would rather not use this option because of need to reboot namenodes.

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: How can I use balancer option "setBalancerBandwidth" with cloudera manager ?

Cloudera Employee

Hi,

 

It you set the balancer bandwidth in Cloudera manager, then when the datanodes are started, they will have that bandwidth seting for balancing operations. However, using the command line it is possible to change the bandwidth while the datanodes or balancer is running, without restarting them. If you do this, you just need to remember that if you restart any datanodes, the bandwidth setting will revert back to that set in Cloudera Manager and you will need to run the command again.

 

To set the balancer bandwidth from the command line and without a restart you can run the following:

 

sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 104857600

 

If you have HA setup for HDFS, the above command may fail, and you should check which is the active namenode and run the command like follows (substituting the correct hostname for activeNamenode below):

 

sudo -u hdfs hdfs dfsadmin -fs hdfs://activeNamenode:8020/ -setBalancerBandwidth 104857600


To check this command worked, the following log entries should appear within the Datanode log files:

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_BALANCERBANDWIDTHUPDATE
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Updating balance throttler bandwidth from 10485760 bytes/s to: 104857600 bytes/s.

1 REPLY 1
Highlighted

Re: How can I use balancer option "setBalancerBandwidth" with cloudera manager ?

Cloudera Employee

Hi,

 

It you set the balancer bandwidth in Cloudera manager, then when the datanodes are started, they will have that bandwidth seting for balancing operations. However, using the command line it is possible to change the bandwidth while the datanodes or balancer is running, without restarting them. If you do this, you just need to remember that if you restart any datanodes, the bandwidth setting will revert back to that set in Cloudera Manager and you will need to run the command again.

 

To set the balancer bandwidth from the command line and without a restart you can run the following:

 

sudo -u hdfs hdfs dfsadmin -setBalancerBandwidth 104857600

 

If you have HA setup for HDFS, the above command may fail, and you should check which is the active namenode and run the command like follows (substituting the correct hostname for activeNamenode below):

 

sudo -u hdfs hdfs dfsadmin -fs hdfs://activeNamenode:8020/ -setBalancerBandwidth 104857600


To check this command worked, the following log entries should appear within the Datanode log files:

INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeCommand action: DNA_BALANCERBANDWIDTHUPDATE
INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Updating balance throttler bandwidth from 10485760 bytes/s to: 104857600 bytes/s.