Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

namenode is not formatted

Highlighted

namenode is not formatted

Expert Contributor

Our server had to be restarted so we are trying to start all of the process :

I keep getting this error. Why is that? How do I fix this? It is urgent as Namenode was on the master node and now the cluster is not working. Please help

Failed to start namenode.
java.io.IOException: NameNode is not formatted.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
8 REPLIES 8
Highlighted

Re: namenode is not formatted

Explorer
Highlighted

Re: namenode is not formatted

Expert Contributor

@Romil Choksi

I did. I have only 2 nodes in my cluster. One with nn and snn. other once has yarn, hdfs and zookeeper. first one is master node.

The entire system went down but the hard disk is safe so there was no data loss for sure.

We are trying to bring back the services one by one. If I run this on ther node, it says

What is the best way I can get this working?

17/04/01 12:15:29 INFO namenode.NameNode: createNameNode [-bootstrapStandby] 17/04/01 12:15:29 ERROR namenode.NameNode: Failed to start namenode. java.io.IOException: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:426) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1504) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606) Caused by: org.apache.hadoop.HadoopIllegalArgumentException: HA is not enabled for this namenode. at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.parseConfAndFindOtherNN(BootstrapStandby.java:378) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:103) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:421) ... 2 more

Highlighted

Re: namenode is not formatted

Expert Contributor

when the system went down, namenode on first server went into standby mode I believe . I could use hadoop fs -ls on the second server then. I can not right now. Any clues?

Highlighted

Re: namenode is not formatted

Explorer

@Simran Kaur Looking at this error, could you please verify if HA is actually enabled for Namenode. Please go through the steps here http://docs.hortonworks.com/HDPDocuments/Ambari-2.2.2.18/bk_ambari-user-guide/content/_namenode_high...

Highlighted

Re: namenode is not formatted

Expert Contributor

@Romil Choksi

Also,

Maximum size of an xattr: 16384
Apr 1, 11:45:18.670 AMINFOorg.apache.hadoop.hdfs.server.common.Storage
Lock on /dfs/nn/in_use.lock acquired by nodename 9392@warehouse.swtched.com
Apr 1, 11:45:18.672 AMWARNorg.apache.hadoop.hdfs.server.namenode.FSNamesystem
Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:212)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1063)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:767)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:609)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:670)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:838)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:817)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1538)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1606)
Apr 1, 11:45:18.684 AMINFOorg.mortbay.log
Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@warehouse.swtched.com:50070
Highlighted

Re: namenode is not formatted

Expert Contributor

I checked. Unfortunately it is not, and for some reason, there is nothing in namenode and secondarynamenode folder although hdfs has data. Does that mean I can't recover it at all?

Re: namenode is not formatted

Explorer

What do you mean by there is nothing in namenode and secondarynamenode folder? Did you try recovering it using the steps described in the HA link ?

Highlighted

Re: namenode is not formatted

@Simran Kaur- How are you trying to restart the Namenodes? Using command line or using Ambari.

Also, as Romil has asked earlier, can you confirm if this HA or non HA cluster:

In HA cluster you will have two Namenodes running as Active and Standy Namenodes. They can switch states to become Active and Standy

In non HA environment there is only one Active Namenode. There is no switching of states. There is a Secondary Namenode, but that is mainly used for checkpointing purposes:

http://blog.madhukaraphatak.com/secondary-namenode---what-it-really-do/

Don't have an account?
Coming from Hortonworks? Activate your account here