HDFS Balancer: Why configure same property?

Question

I am trying to rebalance hdfs with Cloudera Manager 6.3 with HDFS Balancer Document

It says add the same property dfs.datanode.balance.max.concurrent.moves into different section

DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml
Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml

But before adding the property I searched and saw dfs.datanode.balance.max.concurrent.moves was already there. Nevertheless, I did what the document says. After adding properties Cloudera Manager asked me to restart/redeploy stale configurations. Before restart, I saw totally different properties added.

cloudera_manager_rebalancer_cloudera_community_soru.png

I don't understand although we seem to add the same property why different properties are added to hdfs-site.xml?

venkatsambath · Accepted Answer

"although same property (dfs.datanode.balance.max.concurrent.moves) already exists in Cloudera Manager." --> Okay, I assume you are referring to the one highlighted in screenshot below Yes its unnecessary to add dfs.datanode.balance.max.concurrent.moves in Balancer Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml if you had used the "Maximum Concurrent Moves" section. Also note that this "Maximum Concurrent Moves" is scoped only to balancer and not to datanodes. So for datanodes you have to explicitly set it using " DataNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml" Regarding reason for why to add this property both for balancer and datanode is mentioned in my previous comment. Hope that clarifies and let me know if there are further questionsI will raise an internal jira for correcting the document to avoid duplicate entry on balancer safety-valve.

venkatsambath · Answer

@erkansirin78 Let me make sure I understand the issue correctly. By this "

Before restart, I saw totally different properties added." Did you mean the property dfs.datanode.ec.reconstruction.xmts.weight getting added? If yes, then its not getting added instead the preview page is just showing the extra lines prior to the property that you added, only the lines with + sign matters.

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.