Support Questions
Find answers, ask questions, and share your expertise

ambari rebalance hdfs

Expert Contributor

Hi,

can anyone explain why when i run "rebalance hdfs" from Ambari then operation complete success, but really anything was not rebalance? (operation start and immediately end)

Operation under terminal provide true rebalance hdfs.

91680-capture.png

HDFS Rebalance
0 moved / 0 left / 0 being processed

stderr:   /var/lib/ambari-agent/data/errors-3867.txt

stdout:   /var/lib/ambari-agent/data/output-3867.txt

2018-10-11 15:01:39,733 - Using hadoop conf dir: /usr/hdp/2.6.4.0-91/hadoop/conf
2018-10-11 15:01:39,736 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.4.0-91 -> 2.6.4.0-91
2018-10-11 15:01:39,770 - Using hadoop conf dir: /usr/hdp/2.6.4.0-91/hadoop/conf
Starting balancer with threshold = 10
2018-10-11 15:01:39,794 - call['/usr/bin/klist -s /tmp/hdfs_rebalance_cc_85c7072801fc4be10a3eab83067eb7ea'] {'user': 'hdfs'}
2018-10-11 15:01:39,951 - call returned (1, '')
2018-10-11 15:01:39,953 - Execute['/usr/bin/kinit -c /tmp/hdfs_rebalance_cc_85c7072801fc4be10a3eab83067eb7ea -kt /etc/security/keytabs/hdfs.headless.keytab hdfs-testdatalake@?.?'] {'user': 'hdfs'}
Executing command ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/2.6.4.0-91/hadoop/bin'"'"' KRB5CCNAME=/tmp/hdfs_rebalance_cc_85c7072801fc4be10a3eab83067eb7ea ; hdfs --config /usr/hdp/2.6.4.0-91/hadoop/conf balancer -threshold 10'
Checking if the balancer is running ...2018-10-11 15:01:40,168 - ExecuteHadoop['fs -test -e /system/balancer.id'] {'logoutput': True, 'bin_dir': '/usr/hdp/2.6.4.0-91/hadoop/bin', 'user': 'hdfs', 'conf_dir': '/usr/hdp/2.6.4.0-91/hadoop/conf'}
2018-10-11 15:01:40,170 - Execute['hadoop --config /usr/hdp/2.6.4.0-91/hadoop/conf fs -test -e /system/balancer.id'] {'logoutput': True, 'try_sleep': 0, 'environment': {}, 'tries': 1, 'user': 'hdfs', 'path': ['/usr/hdp/2.6.4.0-91/hadoop/bin']}
2018-10-11 15:01:44,507 - Execute['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'export  PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:/usr/hdp/2.6.4.0-91/hadoop/bin'"'"' KRB5CCNAME=/tmp/hdfs_rebalance_cc_85c7072801fc4be10a3eab83067eb7ea ; hdfs --config /usr/hdp/2.6.4.0-91/hadoop/conf balancer -threshold 10''] {'wait_for_finish': False}
The rebalance process has been triggered2018-10-11 15:01:44,516 - File['/tmp/hdfs_rebalance_cc_85c7072801fc4be10a3eab83067eb7ea'] {'action': ['delete']}
2018-10-11 15:01:44,517 - Deleting File['/tmp/hdfs_rebalance_cc_85c7072801fc4be10a3eab83067eb7ea']
2 REPLIES 2

Super Mentor

@Ruslan Fialkovsky

The HDFS Rebalancing operation may take a long time (hours, days) depending on your cluster data thus it's not possible to dynamically predict how long the rebalancing will last so the operation will run at the background. (because this operation is part of 'BACKGROUND_EXECUTION_COMMAND')

So can you please check if you noticed any of the following message in your logs? To be sure that Rebalancing actually was triggered at the background.

Ambari Server log:

# grep 'REBALANCEHDFS' /var/log/ambari-server/ambari-server.log 

In NameNode log check if "balancer" was triggered?

# grep 'balancer.id' /var/log/hadoop/hdfs/hadoop-hdfs-namenode-xxxx.log

.

Expert Contributor

@Jay Kumar SenSharma

Yesterday i run rebalance from ambari and checked ambari server log on 'REBALANCEHDFS' there it was, then i checked namenode log on 'balancer.id' there it wasn't.

The next day when i checked datanode information then rebalance hdfs is only one percent per day.

93427-capture7.png