Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Accessing HDFS shows Name node in safemode

avatar
Rising Star

Hi,

We have configured the NN in HA mode, it is a kerberized cluster. During the weekend we have shutdown all the nodes and today started all the nodes. I've valid TGT issues before issuing hadoop fs -ls commands.

Now when I trying to issue hadoop fs -ls I see the following stack trace : ( Saying it is under safemode). When I checked the HDFS was really in Safemode. Using shell command I made it to move out of safemode.

But still when we issue hadoop fs -ls / on the console I still see that NN are in safe mode.

15/12/21 16:37:13 INFO retry.RetryInvocationHandler: Exception while invoking getFileInfo of class ClientNamenodeProtocolTranslatorPB over Namenode_HOST_2/192.168.1.4:8020 after 1 fail over attempts. Trying to fail over after sleeping for 676ms.

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby
        at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:87)
        at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1872)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:1306)

$ hadoop dfsadmin -safemode get 

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it. 
1 ACCEPTED SOLUTION

avatar
Master Mentor

@Darpan Patel

Looks like failover is still in progress.

Trying to fail over after sleeping for 676ms. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):

Operation category READ is not supported in state standby at..

Restart HDFS components and then try again..

View solution in original post

5 REPLIES 5

avatar
Master Mentor

@Darpan Patel

Looks like failover is still in progress.

Trying to fail over after sleeping for 676ms. org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException):

Operation category READ is not supported in state standby at..

Restart HDFS components and then try again..

avatar
Rising Star

@Neeraj Sabharwal I thought the same but I've hardly 50 MB of data on the cluster. And it is showing this status from last 3 hours.

avatar
Master Mentor

@Darpan Patel Definitely something wrong..You may want to failover again or restart services. Is Ambari in the picture?

avatar
Rising Star
@Neeraj Sabharwal

I started all the datanodes, and then restarted the HDFS master process, this worked!

avatar
Master Mentor

@Darpan Patel Perfect! Please accept one of the answers to close the thread as part of the best practice.