Ambari 2.6 HDFS Re-balance Exits without balancing

[#1]

Hi,

we are experiencing an issue where a all most identical deployment using Ambari 2.6, is able to complete a re-balance of 7 data nodes, in 1 AWS center but not in the second.

the command that we run is

hdfs --loglevel DEBUG balancer -Ddfs.datanode.balance.max.concurrent.moves=20 -Ddfs.balancer.moverThreads=10000 -Ddfs.balancer.dispatcherThreads=10000 -threshold 5.0

we also tried to run this from the Ambari UI, using the same threshold

this is what we get in the log

19/01/29 15:43:58 DEBUG hdfs.HAUtil: No HA service delegation token found for logical URI hdfs://mycluster
19/01/29 15:43:58 INFO balancer.Balancer: Using a threshold of 5.0
19/01/29 15:43:58 INFO balancer.Balancer: namenodes  = [hdfs://mycluster]
19/01/29 15:43:58 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
19/01/29 15:43:58 INFO balancer.Balancer: included nodes = []
19/01/29 15:43:58 INFO balancer.Balancer: excluded nodes = []
19/01/29 15:43:58 INFO balancer.Balancer: source nodes = []
Time Stamp               Iteration#  Bytes Already Moved  Bytes Left To Move  Bytes Being Moved

it then starts to collect info from the nodes

19/01/29 15:44:01 DEBUG balancer.NameNodeConnector: No block has been moved for 1 iterations, maximum notChangedIterations before exit is: 5
Jan 29, 2019 3:44:01 PM           0                  0 B           712.00 GB              20 GB
19/01/29 15:44:10 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
19/01/29 15:44:10 INFO balancer.Balancer: dfs.balancer.moverThreads = 10000 (default=1000)
19/01/29 15:44:10 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 10000 (default=200)
19/01/29 15:44:10 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 20 (default=5)
19/01/29 15:44:10 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
19/01/29 15:44:10 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
19/01/29 15:44:10 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
19/01/29 15:44:10 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)

19/01/29 15:44:10 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => underUtilized
19/01/29 15:44:10 INFO balancer.Balancer: Decided to move 10 GB bytes from xxx.xxx.xxx.149:50010:DISK to xxx.xxx.xxx.34:50010:DISK
19/01/29 15:44:10 INFO balancer.Balancer: Decided to move 10 GB bytes from xxx.xxx.xxx.23:50010:DISK to xxx.xxx.xxx.227:50010:DISK
19/01/29 15:44:10 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: overUtilized => belowAvgUtilized
19/01/29 15:44:10 INFO balancer.Balancer: chooseStorageGroups for SAME_RACK: underUtilized => aboveAvgUtilized
19/01/29 15:44:10 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => underUtilized
19/01/29 15:44:10 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: overUtilized => belowAvgUtilized
19/01/29 15:44:10 INFO balancer.Balancer: chooseStorageGroups for ANY_OTHER: underUtilized => aboveAvgUtilized
19/01/29 15:44:10 INFO balancer.Balancer: Will move 20 GB in this iteration

19/01/29 15:44:12 DEBUG balancer.NameNodeConnector: No block has been moved for 2 iterations, maximum notChangedIterations before exit is: 5
Jan 29, 2019 3:44:12 PM           1                  0 B           712.00 GB              20 GB
19/01/29 15:44:21 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
19/01/29 15:44:21 INFO balancer.Balancer: dfs.balancer.moverThreads = 10000 (default=1000)
19/01/29 15:44:21 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 10000 (default=200)
19/01/29 15:44:21 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 20 (default=5)
19/01/29 15:44:21 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
19/01/29 15:44:21 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
19/01/29 15:44:21 DEBUG sasl.DataTransferSaslUtil: DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection
19/01/29 15:44:21 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
19/01/29 15:44:21 INFO balancer.Balancer: dfs.blocksize = 67108864 (default=134217728)


to end with 

it will do this 5 times end the final result is

19/01/29 15:44:43 DEBUG balancer.NameNodeConnector: No block has been moved for 5 iterations, maximum notChangedIterations before exit is: 5
No block has been moved for 5 iterations. Exiting...
Jan 29, 2019 3:44:43 PM           4                  0 B           712.00 GB              20 GB

for the working cluster we have the same blueprint

does anyone have an idea what could be wrong?

on behave My self @Raymond Honderdors and @Vlad Timofeev thanks in advance