question Re: Error trying to balance disks on node in Archives of Support Questions (Read Only)

Error trying to balance disks on node

velomcb — Fri, 15 Sep 2017 17:42:27 GMT

I'm trying to rebalance hdfs on my data nodes (CDH 5.9.1) after adding a disk. I've followed this blog post: https://blog.cloudera.com/blog/2016/10/how-to-use-the-new-hdfs-intra-datanode-disk-balancer-in-apache-hadoop/ and restarted the cluster. No matter how I format the host (FQDN, hostname only, IP) I always get this response:

[hdfs@ip-10-16-113-100 ~]$ hdfs diskbalancer -plan ip-10-16-113-100
17/09/15 17:34:27 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException

I see no other complaints that I can find, so I'm trying to figure out what I'm doing wrong. Any one gotten this to work that can give me some guidance?

Regards.

Re: Error trying to balance disks on node

velomcb — Mon, 18 Sep 2017 20:44:42 GMT

After some unknown period of time, the command began working. Current guess is that the system has to do a complete block report before the command becomes availible.

Re: Error trying to balance disks on node

Former Member — Mon, 24 Sep 2018 22:10:22 GMT

even I am getting the same error, I see no other error, trying to run the disc plan to run intra disc balancing

error:
hdfs diskbalancer -plan lrcdhdn009
18/09/24 17:01:46 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
18/09/24 17:01:46 INFO block.BlockTokenSecretManager: Setting block keys
18/09/24 17:01:46 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
18/09/24 17:01:46 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

Re: Error trying to balance disks on node

Harsh J — Tue, 25 Sep 2018 02:48:39 GMT

> tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

Please always try with your fully qualified domain names for the target DataNode, so ip-10-16-113-100.ec2.internal, lrcdhdn009.company.com, etc.

Re: Error trying to balance disks on node

Former Member — Tue, 25 Sep 2018 17:13:38 GMT

thanks that worked.

I have an 18 data nodes each capacity of 150 TB, current data storage of each DN is around 14TB whereas 1 DN is 4 TB.

I did disc balancing to equally distribute the data but it didn't help. I am thinking it depends on the math % of storage vs allocated.

is there a way i can distribute the data equally now?

Re: Error trying to balance disks on node

Harsh J — Tue, 25 Sep 2018 22:58:46 GMT

Data balancing uses the relative measure of percentage for heterogenous
nodes so that it scales based on actual capacities. Are you looking to
balance by byte count instead? That doesn't sound like a good idea for such
a wide difference in space. Could you help further explain your goal here?

Re: Error trying to balance disks on node

Suhas_Ganorkar — Tue, 07 Nov 2023 11:07:23 GMT

1) Remember all data nodes should be healthy to perform this activity, if any disk failure is there, it will throw error like this;
ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException

2) ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException): Disk Balancer is not enabled.
add property in HDFS configuration >> dfs.disk.balancer.enabled << under HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml, value = true

3) ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied.
login through datanode keytab;
cd /var/run/cloudera-scm-agent/process && find . -type f -iname "hdfs.keytab"
kinit -k -t ./52821-hdfs-DATANODE/hdfs.keytab -p hdfs/hostname@SOME_REALM.COM