Error trying to balance disks on node

[#1]

I'm trying to rebalance hdfs on my data nodes (CDH 5.9.1) after adding a disk.  I've followed this blog post: https://blog.cloudera.com/blog/2016/10/how-to-use-the-new-hdfs-intra-datanode-disk-balancer-in-apach... and restarted the cluster.  No matter how I format the host (FQDN, hostname only, IP) I always get this response:

[hdfs@ip-10-16-113-100 ~]$ hdfs diskbalancer -plan ip-10-16-113-100
17/09/15 17:34:27 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException

I see no other complaints that I can find, so I'm trying to figure out what I'm doing wrong.  Any one gotten this to work that can give me some guidance?

Regards.

[#2 · Accepted solution]

After some unknown period of time, the command began working.  Current guess is that the system has to do a complete block report before the command becomes availible.

[#3]

even I am getting the same error, I see no other error, trying to run the disc plan to run intra disc balancing

error:
hdfs diskbalancer -plan lrcdhdn009
18/09/24 17:01:46 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
18/09/24 17:01:46 INFO block.BlockTokenSecretManager: Setting block keys
18/09/24 17:01:46 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
18/09/24 17:01:46 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

[#4]

> tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

Please always try with your fully qualified domain names for the target DataNode, so ip-10-16-113-100.ec2.internal, lrcdhdn009.company.com, etc.

[#5]

thanks that worked.

I have an 18 data nodes each capacity of 150 TB, current data storage of each DN is around 14TB whereas 1 DN is 4 TB.

I did disc balancing to equally distribute the data but it didn't help. I am thinking it depends on the math % of storage vs allocated.

is there a way i can distribute the data equally now?

[#6]

Data balancing uses the relative measure of percentage for heterogenous
nodes so that it scales based on actual capacities. Are you looking to
balance by byte count instead? That doesn't sound like a good idea for such
a wide difference in space. Could you help further explain your goal here?

[#7]

1) Remember all data nodes should be healthy to perform this activity, if any disk failure is there, it will throw error like this;
ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException

2) ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException): Disk Balancer is not enabled.
add property in HDFS configuration >> dfs.disk.balancer.enabled << under HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml, value = true

3) ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied.
login through datanode keytab;
cd /var/run/cloudera-scm-agent/process && find . -type f -iname "hdfs.keytab"
kinit -k -t ./52821-hdfs-DATANODE/hdfs.keytab -p hdfs/hostname@SOME_REALM.COM