Cannot restart namenode: Failed to load an FSImage file

[#1]

I have Cluster Express with 1 master role, 1 management role and 6 worker roles. My Namenode and Master showed not running and I cannot restart the Namenode. Error below:

2015-11-29 19:27:32,758 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
java.io.IOException: Failed to load an FSImage file!
	at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:657)
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:275)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:880)
	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:639)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:500)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:556)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:721)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:705)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1355)
	at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1421)

I ssh'ed into the master role VM and tried to fix it manually:

hdfs namenode -recover

But it is still the same. Let me know where I should troubleshoot next. Thanks

[#2]

Hi hawkphil,

What did you actually do in the cluster before you face the error?

Which version are you running on?

Dice.

[#3]

I am on the latest version, whichever I downaloded last week v5.5 and I installed in vSphere

My lab got an outage so the entire cluster went off disruptively. That was the main reason causing all weird thing. I didn't have too much valuable data in there yet

[#4]

Just a gentle nudge...

Any suggestion please?

[#5]

Check if your namenode.name.dir was pointing to a directory where fsimage doesn't exist, make it pointing to the directory where previous fsimage files exit.

[#6]

It is pointing to /dfs/nn and there is ./current folder in there
I think it's corrupted. I replaced with /dfs/snn/current and it's still not working
Then I use Cloudera Manager to redeploy the Namenode / Secondary Namenode to some other VMs and it seems to work .. for now.
Strange.