Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HA for NameNode doesn't work

Solved Go to solution

HA for NameNode doesn't work

New Contributor

We are trying to launch HA for NameNode and ResourceManager by using blueprint. We are using HDP 2.3, Ambari 2.1 and 2.2. There are two issues:

1. Namenodes and DataNodes fail to start because of "too small initial heap size" exception. After some research we figure out that several default configs in hadoop-env.sh file don't have 'm' letter (which means 'megabyte'), namely: XX:NewSize={{namenode_opt_newsize}}, XX:MaxNewSize={{namenode_opt_maxnewsize}}, Xms{{namenode_heapsize}}, Xmx{{namenode_heapsize}}, Xmx{{dtnode_heapsize}}. This problem occurs only when we enable HA. Without HA all these configs have 'm' and all works fine.

2. We explicitly setup wrong default configs that pointed above. But we get this situation: NameNodes and ZKFCs are started successfully but after this they are suddenly stopped. In logs we have "NameNode is not formatted" exception.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: HA for NameNode doesn't work

New Contributor

We have found the problem. HA for Namenode works fine. But when we are deploying cluster using blueprint only for Resourcemanager HA then Datanodes and Namenode can't to start because of "too small initial heap size" and "too small heap size" exceptions. After some research we figure out that we had missed several configs in our blueprint: yarn.resourcemanager.webapp.address.rm, yarn.resourcemanager.webapp.https.address.rm, fs.defaultFS, hadoop.registry.zk.quorum. Some of these configs are missing in the example for Resourcemanager HA in https://cwiki.apache.org/confluence/display/AMBARI/Blueprint+Support+for+HA+Clusters

5 REPLIES 5

Re: HA for NameNode doesn't work

@Michael Ionkin

https://cwiki.apache.org/confluence/display/AMBARI/Blueprint+Support+for+HA+Clusters

You may want to reinstall with correct steps/configs otherwise, you have to format the namenode manually to bring up NN.

ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.io.IOException: NameNode is not formatted

hadoop namenode -format

Highlighted

Re: HA for NameNode doesn't work

Important! Only format the Namenode if you do not have any data in your cluster!

Re: HA for NameNode doesn't work

New Contributor

@Neeraj Sabharwal, thank you for reply. We are creating this cluster and blueprint according to steps in the link you've posted above. The question is why we should manually format namenode? If we create cluster from blueprint then it should start without any extra actions from our side, shouldn't it?

Also we cannot understand why defaults configs have incorrect properties if we enable HA (issue 1 in my initial question)

Maybe this bug https://hortonworks.jira.com/browse/BUG-40852 is our case?

Re: HA for NameNode doesn't work

@Michael Ionkin You mentioned that there were errors during deployment and confusion in configuration.

https://community.hortonworks.com/questions/11789/ambari-blueprints-and-namenode-metadata.html

See this

Re: HA for NameNode doesn't work

New Contributor

We have found the problem. HA for Namenode works fine. But when we are deploying cluster using blueprint only for Resourcemanager HA then Datanodes and Namenode can't to start because of "too small initial heap size" and "too small heap size" exceptions. After some research we figure out that we had missed several configs in our blueprint: yarn.resourcemanager.webapp.address.rm, yarn.resourcemanager.webapp.https.address.rm, fs.defaultFS, hadoop.registry.zk.quorum. Some of these configs are missing in the example for Resourcemanager HA in https://cwiki.apache.org/confluence/display/AMBARI/Blueprint+Support+for+HA+Clusters

Don't have an account?
Coming from Hortonworks? Activate your account here