namenode is not formatted

[#1]

Our server had to be restarted so we are trying to start all of the process :

I keep getting this error. Why is that? How do I fix this? It is urgent as Namenode was on the master node and now the cluster is not working. Please help

Failed to start namenode.
java.io.IOException: NameNode is not formatted.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)

[#2]

@Simran Kaur

Please try out the steps described in this support kb

https://community.hortonworks.com/content/supportkb/49454/restarting-standby-namenode-fails-to-start...

[#3]

@Romil Choksi

I did. I have only 2 nodes in my cluster. One with nn and snn. other once has yarn, hdfs and zookeeper. first one is master node.

The entire system went down but the hard disk is safe so there was no data loss for sure.

We are trying to bring back the services one by one. If I run this on ther node, it says

What is the best way I can get this working?

17/04/01 12:15:29 INFO namenode.NameNode: createNameNode [-bootstrapStandby] 17/04/01 12:15:29 ERROR namenode.NameNode: Failed to start namenode. java.io.IOException: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:426) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1504) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606) Caused by: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.parseConfAndFindOtherNN(BootstrapStandby.java:378) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:421) ... 2 more

[#4]

when the system went down, namenode on first server went into standby mode I believe . I could use hadoop fs -ls on the second server then. I can not right now. Any clues?

[#5]

@Simran Kaur Looking at this error, could you please verify if HA is actually enabled for Namenode. Please go through the steps here http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.18/bk_ambari-user-guide/content/_namenode_high...

[#6]

@Romil Choksi

Also,

Maximum size of an xattr: 16384
Apr 1, 11:45:18.670 AM	INFO	org.apache.hadoop.hdfs.server.common.Storage	Lock on /dfs/nn/in_use.lock acquired by nodename 9392@warehouse.swtched.com
Apr 1, 11:45:18.672 AM	WARN	org.apache.hadoop.hdfs.server.namenode.FSNamesystem	Encountered exception loading fsimage java.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
Apr 1, 11:45:18.684 AM	INFO	org.mortbay.log	Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@warehouse.swtched.com:50070

[#7]

I checked. Unfortunately it is not, and for some reason, there is nothing in namenode and secondarynamenode folder although hdfs has data. Does that mean I can't recover it at all?

[#8]

What do you mean by there is nothing in namenode and secondarynamenode folder? Did you try recovering it using the steps described in the HA link ?

[#9]

@Simran Kaur- How are you trying to restart the Namenodes? Using command line or using Ambari.

Also, as Romil has asked earlier, can you confirm if this HA or non HA cluster:

In HA cluster you will have two Namenodes running as Active and Standy Namenodes. They can switch states to become Active and Standy

In non HA environment there is only one Active Namenode. There is no switching of states. There is a Secondary Namenode, but that is mainly used for checkpointing purposes:

http://blog.madhukaraphatak.com/secondary-namenode---what-it-really-do/

[#10]

@simran_k  

format your namenode by using below command:

$ bin/hdfs namenode -format