Support Questions
Find answers, ask questions, and share your expertise

Accessing HDFS shows Name node in safemode

Solved Go to solution
Highlighted

Accessing HDFS shows Name node in safemode

Explorer

Hi,

We have configured the NN in HA mode, it is a kerberized cluster. During the weekend we have shutdown all the nodes and today started all the nodes. I've valid TGT issues before issuing hadoop fs -ls commands.

Now when I trying to issue hadoop fs -ls I see the following stack trace : ( Saying it is under safemode). When I checked the HDFS was really in Safemode. Using shell command I made it to move out of safemode.

But still when we issue hadoop fs -ls / on the console I still see that NN are in safe mode.

15/12/21 16:37:13 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over Namenode_HOST_2/192.168.1.4:8020 after 1 fail over attempts. Trying to fail over after sleeping for 676ms.

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1872)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1306)

$ hadoop dfsadmin -safemode get 

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it. 
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Accessing HDFS shows Name node in safemode

@Darpan Patel

Looks like failover is still in progress.

Trying to fail over after sleeping for 676ms. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):

Operation category READ is not supported in state standby at..

Restart HDFS components and then try again..

View solution in original post

5 REPLIES 5
Highlighted

Re: Accessing HDFS shows Name node in safemode

@Darpan Patel

Looks like failover is still in progress.

Trying to fail over after sleeping for 676ms. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):

Operation category READ is not supported in state standby at..

Restart HDFS components and then try again..

View solution in original post

Highlighted

Re: Accessing HDFS shows Name node in safemode

Explorer

@Neeraj Sabharwal I thought the same but I've hardly 50 MB of data on the cluster. And it is showing this status from last 3 hours.

Highlighted

Re: Accessing HDFS shows Name node in safemode

@Darpan Patel Definitely something wrong..You may want to failover again or restart services. Is Ambari in the picture?

Re: Accessing HDFS shows Name node in safemode

Explorer
@Neeraj Sabharwal

I started all the datanodes, and then restarted the HDFS master process, this worked!

Highlighted

Re: Accessing HDFS shows Name node in safemode

@Darpan Patel Perfect! Please accept one of the answers to close the thread as part of the best practice.