Resolving HDFS NameNode Standby Errors: Enabling Automatic Failover and ZKFC

Question

I currently have one namenode in a 'stopped' state due to a node failure. I am unable to access any data or services on the cluster, as this was the main namenode.

However, there is a second namenode that I am hoping can be used to recover. I have been working on the issue in this thread and currently I all hdfs instances started except for the bad namenode. This seems to have improved the situation as far as node health status but I still can't access any data.

Here is the relevant command and error:

ubuntu@ip-10-0-0-154:~/backup/data1$ hdfs dfs -ls hdfs://10.0.0.154:8020/ ls: Operation category READ is not supported in state standby

In the previous thread, I also pointed out that there was the option to enable automatic failure in CM. I am wondering if that is the best course of action right now. Any help is greatly appreciated.

epowell · Accepted Answer

As noted in the previous reply, I did not have any nodes with the Failover Controller role. Importantly, I also had not enabled Automatic Failover despite running in an HA configuration.

I went ahead and added the Failover Controller role to both namenodes - the good one and the bad one.

After that, I attempted enable the Automatic Failover . To do that, however, I needed to first start Zookeeper.

At that point, If I recall correctly, the other namenode was still not active but I then restarted the entire cluster and the automatic failover kicked in, using the other namenode as the active one and leaving the bad namenode in a stopped state.

saranvisa · Answer

@epowell

The issue might be related to the below jira which is opened a long back still in open status

https://issues.apache.org/jira/browse/HDFS-3447

as an alternate way to connect to hdfs, go to hdfs-site.xml and get dfs.nameservices and try to connect to hdfs using namespace as follows, it may help you

hdfs://<ClusterName>-ns/<hdfs_path>

Note: I didn't get a chance to explore this... also not sure how it will respond in old cdh version

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.