Accessing HDFS shows Name node in safemode

[#1]

Hi,

We have configured the NN in HA mode, it is a kerberized cluster. During the weekend we have shutdown all the nodes and today started all the nodes. I've valid TGT issues before issuing hadoop fs -ls commands.

Now when I trying to issue hadoop fs -ls I see the following stack trace : ( Saying it is under safemode). When I checked the HDFS was really in Safemode. Using shell command I made it to move out of safemode.

But still when we issue hadoop fs -ls / on the console I still see that NN are in safe mode.

15/12/21 16:37:13 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over Namenode_HOST_2/192.168.1.4:8020 after 1 fail over attempts. Trying to fail over after sleeping for 676ms.

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1872)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1306)

$ hadoop dfsadmin -safemode get 

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it. 

[#2 · Accepted solution]

@Darpan Patel

Looks like failover is still in progress.

Trying to fail over after sleeping for 676ms. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):

Operation category READ is not supported in state standby at..

Restart HDFS components and then try again..

[#3]

@Neeraj Sabharwal I thought the same but I've hardly 50 MB of data on the cluster. And it is showing this status from last 3 hours.

[#4]

@Darpan Patel Definitely something wrong..You may want to failover again or restart services. Is Ambari in the picture?

[#5]

@Neeraj Sabharwal

I started all the datanodes, and then restarted the HDFS master process, this worked!

[#6]

@Darpan Patel Perfect! Please accept one of the answers to close the thread as part of the best practice.