question HDP HDFS balancer CLI not found in Support Questions

HDP HDFS balancer CLI not found

wazzu62 — Tue, 14 Jun 2022 20:25:27 GMT

We had an HDP 2.5.3.0 cluster that had an HDFS balancer CLI command that is not found with HDP 3.1.5.0. I am adding new nodes the the 3.1.5 cluster and it appears there is some HDFS balancing going on, albeit slowly. Did the CLI go away with 3.1.5.0 or am I missing something?

Re: HDP HDFS balancer CLI not found

Tylenol — Wed, 15 Jun 2022 05:14:14 GMT

Hello @wazzu62 , Kindly share what is the error message that you get while you run the hdfs balancer command as we haven't removed the CLI command on HDP 3.1.5.0.

Re: HDP HDFS balancer CLI not found

wazzu62 — Wed, 15 Jun 2022 13:26:28 GMT

Operator error, I forgot the balancer was an hdfs command and not something in /usr/bin

Re: HDP HDFS balancer CLI not found

Tylenol — Wed, 15 Jun 2022 14:57:38 GMT

ah ! Can you try to run the below HDFS balancer command , The below command would move the blocks at a decent pace and would not affect the existing jobs

nohup hdfs balancer -Ddfs.balancer.moverThreads=5000 -Ddfs.datanode.balance.max.concurrent.moves=20 -Ddfs.datanode.balance.bandwidthPerSec=10737418240 -Ddfs.balancer.dispatcherThreads=200 -Ddfs.balancer.max-size-to-move=100737418240 -threshold 10 1>/home/hdfs/balancer/balancer-out_$(date +"%Y%m%d%H%M%S").log 2>/home/hdfs/balancer/balancer-err_$(date +"%Y%m%d%H%M%S").log

you can also refer to the below doc if you need any tuning
https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/data-storage/content/balancer_commands.html