Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

namenode is not formatted

avatar
Expert Contributor

Our server had to be restarted so we are trying to start all of the process :

I keep getting this error. Why is that? How do I fix this? It is urgent as Namenode was on the master node and now the cluster is not working. Please help

Failed to start namenode.
java.io.IOException: NameNode is not formatted.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
9 REPLIES 9

avatar
Contributor

avatar
Expert Contributor

@Romil Choksi

I did. I have only 2 nodes in my cluster. One with nn and snn. other once has yarn, hdfs and zookeeper. first one is master node.

The entire system went down but the hard disk is safe so there was no data loss for sure.

We are trying to bring back the services one by one. If I run this on ther node, it says

What is the best way I can get this working?

17/04/01 12:15:29 INFO namenode.NameNode: createNameNode [-bootstrapStandby] 17/04/01 12:15:29 ERROR namenode.NameNode: Failed to start namenode. java.io.IOException: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:426) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1504) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606) Caused by: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.parseConfAndFindOtherNN(BootstrapStandby.java:378) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:421) ... 2 more

avatar
Expert Contributor

when the system went down, namenode on first server went into standby mode I believe . I could use hadoop fs -ls on the second server then. I can not right now. Any clues?

avatar
Contributor

@Simran Kaur Looking at this error, could you please verify if HA is actually enabled for Namenode. Please go through the steps here http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.18/bk_ambari-user-guide/content/_namenode_high...

avatar
Expert Contributor

@Romil Choksi

Also,

Maximum size of an xattr: 16384
Apr 1, 11:45:18.670 AMINFOorg.apache.hadoop.hdfs.server.common.Storage
Lock on /dfs/nn/in_use.lock acquired by nodename 9392@warehouse.swtched.com
Apr 1, 11:45:18.672 AMWARNorg.apache.hadoop.hdfs.server.namenode.FSNamesystem
Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
Apr 1, 11:45:18.684 AMINFOorg.mortbay.log
Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@warehouse.swtched.com:50070

avatar
Expert Contributor

I checked. Unfortunately it is not, and for some reason, there is nothing in namenode and secondarynamenode folder although hdfs has data. Does that mean I can't recover it at all?

avatar
Contributor

What do you mean by there is nothing in namenode and secondarynamenode folder? Did you try recovering it using the steps described in the HA link ?

avatar

@Simran Kaur- How are you trying to restart the Namenodes? Using command line or using Ambari.

Also, as Romil has asked earlier, can you confirm if this HA or non HA cluster:

In HA cluster you will have two Namenodes running as Active and Standy Namenodes. They can switch states to become Active and Standy

In non HA environment there is only one Active Namenode. There is no switching of states. There is a Secondary Namenode, but that is mainly used for checkpointing purposes:

http://blog.madhukaraphatak.com/secondary-namenode---what-it-really-do/

avatar
New Contributor

@simran_k  

format your namenode by using below command:

 

$ bin/hdfs namenode -format