HA for NameNode doesn't work

[#1]

We are trying to launch HA for NameNode and ResourceManager by using blueprint. We are using HDP 2.3, Ambari 2.1 and 2.2. There are two issues:

1. Namenodes and DataNodes fail to start because of "too small initial heap size" exception. After some research we figure out that several default configs in hadoop-env.sh file don't have 'm' letter (which means 'megabyte'), namely: XX:NewSize={{namenode_opt_newsize}}, XX:MaxNewSize={{namenode_opt_maxnewsize}}, Xms{{namenode_heapsize}}, Xmx{{namenode_heapsize}}, Xmx{{dtnode_heapsize}}. This problem occurs only when we enable HA. Without HA all these configs have 'm' and all works fine.

2. We explicitly setup wrong default configs that pointed above. But we get this situation: NameNodes and ZKFCs are started successfully but after this they are suddenly stopped. In logs we have "NameNode is not formatted" exception.

[#2]

@Michael Ionkin

https://cwiki.apache.org/confluence/display/AMBARI/Blueprint+Support+for+HA+Clusters

You may want to reinstall with correct steps/configs otherwise, you have to format the namenode manually to bring up NN.

ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.io.IOException: NameNode is not formatted

hadoop namenode -format

[#3]

Important! Only format the Namenode if you do not have any data in your cluster!

[#4]

@Neeraj Sabharwal, thank you for reply. We are creating this cluster and blueprint according to steps in the link you've posted above. The question is why we should manually format namenode? If we create cluster from blueprint then it should start without any extra actions from our side, shouldn't it?

Also we cannot understand why defaults configs have incorrect properties if we enable HA (issue 1 in my initial question)

Maybe this bug https://hortonworks.jira.com/browse/BUG-40852 is our case?

[#5]

@Michael Ionkin You mentioned that there were errors during deployment and confusion in configuration.

https://community.hortonworks.com/questions/11789/ambari-blueprints-and-namenode-metadata.html

See this

[#6 · Accepted solution]

We have found the problem. HA for Namenode works fine. But when we are deploying cluster using blueprint only for Resourcemanager HA then Datanodes and Namenode can't to start because of "too small initial heap size" and "too small heap size" exceptions. After some research we figure out that we had missed several configs in our blueprint: yarn.resourcemanager.webapp.address.rm, yarn.resourcemanager.webapp.https.address.rm, fs.defaultFS, hadoop.registry.zk.quorum. Some of these configs are missing in the example for Resourcemanager HA in https://cwiki.apache.org/confluence/display/AMBARI/Blueprint+Support+for+HA+Clusters