Even when i ran balancer, load one data node is 84%. What should be the reason?

[#1]

@Ram D

1. Can you attached screenshot of namenode ui [showing datanode utilization]

2. Can you paste output $hdfs dfsadmin -report

3. top command output from the datanode which has load.

[#2]

Please find the attachments.nnui1.pngtop-and-dfsadmin-report.txt

[#3]

@Ram D

What is the value of dfs.balance.bandwidthPerSec? What version of Hadoop are you using? There is also protection built in to limit how many blocks can be moved. This is important because you may have other processes running and moving all the data fast will impact those processes. What is the value of dfs.datanode.balance.max.concurrent.moves? A low value will impact how many threads can concurrently move data.

You can increase concurrency but be mindful of the bandwidth you will use and that it is less than your configured dfs.balance.bandwidthPerSec.

[#4 · Accepted solution]

The question is unclear to me but I recommend reading the following three blog posts carefully as they go into great detail about balancer basics, configuration and best practices:

https://community.hortonworks.com/articles/43615/hdfs-balancer-1-100x-performance-improvement.html

https://community.hortonworks.com/articles/43849/hdfs-balancer-2-configurations-cli-options.html

https://community.hortonworks.com/articles/44148/hdfs-balancer-3-cluster-balancing-algorithm.html