Reply
New Contributor
Posts: 3
Registered: ‎10-01-2015

Failed to start namenode - Directory is in an inconsistent state:

[ Edited ]

Hey all, 

 

I am having trouble with my Cluster Set-up (installing 5.4.7).  The 4th step "Checking if the name directories of the NameNode are empty. Formatting HDFS only if empty" is failing.

 

Reviewing the logs, I get the following error:

 

Failed to start namenode.
org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /u01/dfs/nn is in an inconsistent state: storage directory does not exist or is not accessible.

 

15/10/01 09:29:02 WARN namenode.NameNode: Encountered exception during format:
java.io.IOException: Cannot create directory /u01/dfs/nn/current

 

So I thought this was a permissions thing, but after chaning permissions and also ownership on the directory in question, the setup still fails.  

 

Has anyone see this before, or know how to solve?

 

Much apprecaite, take care.

 

-Salah Ayoubi

New Contributor
Posts: 3
Registered: ‎10-01-2015

Re: Failed to start namenode - Directory is in an inconsistent state:

Figured out my issue...rookie mistake on my part.  The parent directory had the permissions problem.  Talking to the duck works... :)

Posts: 882
Kudos: 102
Solutions: 54
Registered: ‎04-06-2015

Re: Failed to start namenode - Directory is in an inconsistent state:

Congratulations on solving your problem and thanks for updating the post too. :)



Cy Jervis, Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

New Contributor
Posts: 3
Registered: ‎10-01-2015

Re: Failed to start namenode - Directory is in an inconsistent state:

Thanks!! Much appreciated!
vga
Explorer
Posts: 14
Registered: ‎09-30-2016

Re: Failed to start namenode - Directory is in an inconsistent state:

Hi

 

I'm getting similar problem while installing cloudera manager on ubuntu. I'm installing with cloudera-manager-installer.bin

 

16/10/15 15:48:18 INFO namenode.NNConf: XAttrs enabled? true
16/10/15 15:48:18 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/10/15 15:48:18 INFO namenode.FSImage: Allocated new BlockPoolId: BP-466403930-192.168.0.5-1476568098276
16/10/15 15:48:18 WARN namenode.NameNode: Encountered exception during format:
java.io.IOException: Cannot create directory /dfs/nn/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:343)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:548)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:569)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:159)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1046)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1484)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1611)
16/10/15 15:48:18 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Cannot create directory /dfs/nn/current
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:343)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:548)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:569)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:159)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1046)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1484)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1611)
16/10/15 15:48:18 INFO util.ExitUtil: Exiting with status 1
16/10/15 15:48:18 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at venu-HP-Pavilion-Sleekbook-14-PC.exam.com/192.168.0.5

Contributor
Posts: 52
Registered: ‎10-19-2015

Re: Failed to start namenode - Directory is in an inconsistent state:

Check what is the admin user of hdfs service and then check if this has permission to create /dfs/nn/current

vga
Explorer
Posts: 14
Registered: ‎09-30-2016

Re: Failed to start namenode - Directory is in an inconsistent state:

Thank you. I just moved to cloudera quick VM installation since it is giving me very hard time on Ubuntu.

Highlighted
New Contributor
Posts: 5
Registered: ‎05-18-2017

Re: Failed to start namenode - Directory is in an inconsistent state:

Hi,

 

I am seeing a similar kind of issue on my namenode servers.

When I tried to restart my namenode, it failed initially with permission issues on /data1/dfs/nn.

The group identifier didn't exist in the list of groups I had.

So, I changed the permissions on the /data1/dfs/nn and restarted it.

This fixed the issue with /data1/dfs/nn but later starting throwing similar kind of errors with /data2/dfs/nn.

I repeated the same steps on /data2/dfs/nn hoping that this will fix the issue.But no luck.

For the same name node, why is it different for the data dirs.

Any pointers ?

Announcements