Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error trying to balance disks on node

avatar
Explorer

I'm trying to rebalance hdfs on my data nodes (CDH 5.9.1) after adding a disk.  I've followed this blog post: https://blog.cloudera.com/blog/2016/10/how-to-use-the-new-hdfs-intra-datanode-disk-balancer-in-apach... and restarted the cluster.  No matter how I format the host (FQDN, hostname only, IP) I always get this response:

 

[hdfs@ip-10-16-113-100 ~]$ hdfs diskbalancer -plan ip-10-16-113-100
17/09/15 17:34:27 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException

I see no other complaints that I can find, so I'm trying to figure out what I'm doing wrong.  Any one gotten this to work that can give me some guidance?

 

Regards.

1 ACCEPTED SOLUTION

avatar
Explorer

After some unknown period of time, the command began working.  Current guess is that the system has to do a complete block report before the command becomes availible.

View solution in original post

6 REPLIES 6

avatar
Explorer

After some unknown period of time, the command began working.  Current guess is that the system has to do a complete block report before the command becomes availible.

avatar
Contributor
even I am getting the same error, I see no other error, trying to run the disc plan to run intra disc balancing

error:
hdfs diskbalancer -plan lrcdhdn009
18/09/24 17:01:46 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
18/09/24 17:01:46 INFO block.BlockTokenSecretManager: Setting block keys
18/09/24 17:01:46 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
18/09/24 17:01:46 ERROR tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

avatar
Mentor
> tools.DiskBalancer: java.lang.IllegalArgumentException: Unable to find the specified node. lrcdhdn009

Please always try with your fully qualified domain names for the target DataNode, so ip-10-16-113-100.ec2.internal, lrcdhdn009.company.com, etc.

avatar
Contributor

thanks that worked.

 

I have an 18 data nodes each capacity of 150 TB, current data storage of each DN is around 14TB whereas 1 DN is 4 TB.

 

I did disc balancing to equally distribute the data but it didn't help. I am thinking it depends on the math % of storage vs allocated.

is there a way i can distribute the data equally now?

 

 

avatar
Mentor
Data balancing uses the relative measure of percentage for heterogenous
nodes so that it scales based on actual capacities. Are you looking to
balance by byte count instead? That doesn't sound like a good idea for such
a wide difference in space. Could you help further explain your goal here?

avatar
New Contributor

1) Remember all data nodes should be healthy to perform this activity, if any disk failure is there, it will throw error like this;
ERROR tools.DiskBalancerCLI: java.lang.IllegalArgumentException


2) ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.diskbalancer.DiskBalancerException): Disk Balancer is not enabled.
add property in HDFS configuration >> dfs.disk.balancer.enabled << under HDFS Service Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml, value = true


3) ERROR tools.DiskBalancerCLI: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied.
login through datanode keytab;
cd /var/run/cloudera-scm-agent/process && find . -type f -iname "hdfs.keytab"
kinit -k -t ./52821-hdfs-DATANODE/hdfs.keytab -p hdfs/hostname@SOME_REALM.COM