question Re: HA for NameNode doesn't work in Support Questions

HA for NameNode doesn't work

mionkin — Tue, 01 Mar 2016 02:21:46 GMT

We are trying to launch HA for NameNode and ResourceManager by using blueprint. We are using HDP 2.3, Ambari 2.1 and 2.2. There are two issues:

1. Namenodes and DataNodes fail to start because of "too small initial heap size" exception. After some research we figure out that several default configs in hadoop-env.sh file don't have 'm' letter (which means 'megabyte'), namely: XX:NewSize={{namenode_opt_newsize}}, XX:MaxNewSize={{namenode_opt_maxnewsize}}, Xms{{namenode_heapsize}}, Xmx{{namenode_heapsize}}, Xmx{{dtnode_heapsize}}. This problem occurs only when we enable HA. Without HA all these configs have 'm' and all works fine.

2. We explicitly setup wrong default configs that pointed above. But we get this situation: NameNodes and ZKFCs are started successfully but after this they are suddenly stopped. In logs we have "NameNode is not formatted" exception.

Re: HA for NameNode doesn't work

nsabharwal — Tue, 01 Mar 2016 09:44:08 GMT

@Michael Ionkin

https://cwiki.apache.org/confluence/display/AMBARI/Blueprint+Support+for+HA+Clusters

You may want to reinstall with correct steps/configs otherwise, you have to format the namenode manually to bring up NN.

ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.io.IOException: NameNode is not formatted

hadoop namenode -format

Re: HA for NameNode doesn't work

jstraub — Tue, 01 Mar 2016 13:44:55 GMT

Important! Only format the Namenode if you do not have any data in your cluster!

Re: HA for NameNode doesn't work

mionkin — Tue, 01 Mar 2016 18:49:37 GMT

@Neeraj Sabharwal, thank you for reply. We are creating this cluster and blueprint according to steps in the link you've posted above. The question is why we should manually format namenode? If we create cluster from blueprint then it should start without any extra actions from our side, shouldn't it?

Also we cannot understand why defaults configs have incorrect properties if we enable HA (issue 1 in my initial question)

Maybe this bug https://hortonworks.jira.com/browse/BUG-40852 is our case?

Re: HA for NameNode doesn't work

nsabharwal — Tue, 01 Mar 2016 18:59:14 GMT

@Michael Ionkin You mentioned that there were errors during deployment and confusion in configuration.

https://community.hortonworks.com/questions/11789/ambari-blueprints-and-namenode-metadata.html

See this

Re: HA for NameNode doesn't work

mionkin — Sat, 05 Mar 2016 00:00:56 GMT

We have found the problem. HA for Namenode works fine. But when we are deploying cluster using blueprint only for Resourcemanager HA then Datanodes and Namenode can't to start because of "too small initial heap size" and "too small heap size" exceptions. After some research we figure out that we had missed several configs in our blueprint: yarn.resourcemanager.webapp.address.rm, yarn.resourcemanager.webapp.https.address.rm, fs.defaultFS, hadoop.registry.zk.quorum. Some of these configs are missing in the example for Resourcemanager HA in https://cwiki.apache.org/confluence/display/AMBARI/Blueprint+Support+for+HA+Clusters