In our cluster Namenode switchover is happening everyday . Can anyone please help me in knowing which logs to check for the root cause of switchover which is happening again and again.
Unfortunately, you haven't described your cluster setup but my assumption is that you have 3 Zk's in your HA implementation. There are two components deployed to Hadoop HDFS for implementing Automatic Failover.
These two components are-
The ZKFC is the ZooKeeper client, who is also responsible for managing and monitoring the NameNode state. ZKFC is a client that runs on all nodes on the Hadoop cluster, which is running NameNode.
These 2 components are responsible for:
ZKFC is accountable for health monitoring heart beating the NameNode with health-check commands periodically. As long as the NameNode responds with a healthy status timely, it considers the NameNode as healthy. In this case, if the NameNode got crashed, froze, or entered an unhealthy state, then it marks the NameNode as unhealthy.
ZooKeeper session management
It is also responsible for the session management with ZooKeeper. The ZKFC maintains a session open in the ZooKeeper when the local Namenode is healthy. Also, if the Local NameNode is the active NameNode, then with the session, it also holds a special lock “znode”. This lock uses ZooKeeper support for the ”ephemeral” nodes. Thus, if the session gets expires, the lock node will be deleted automatically.
When the local Namenode is healthy and ZKFC finds that no other NameNode acquires the lock “znode”, then it will try by itself to acquire the lock. If it gets successful in obtaining the lock, then ZKFC has won the election, and now it is responsible for running the failover to make its local NameNode active. The failover process run by the ZKFC is similar to the failover process run by the manual failover described in the NameNode High Availability article.
2. ZooKeeper quorum
A ZK quorum is a highly available service for maintaining little amounts of coordination data. It notifies the clients about the changes in that data. It monitors clients for the failures.
The HDFS implementation of automatic failover depends on ZooKeeper for the following things:
How does it detect NN Failure each NameNode machine in the Hadoop cluster maintains a persistent session in the ZooKeeper. If any of the machines crashes, then the ZooKeeper session maintained will get expire—zooKeeper than reveal to all the other NameNodes to start the failover process.
To exclusively select the active NameNode, ZooKeeper provides a simple mechanism. In the case of active NameNode failure, another standby NameNode may take the special exclusive lock in the ZooKeeper, stating that it should become the next active NameNode.
Taking all the above into account, the first component logs to check are Zk and NN,
My suspicion is you have issues with the Namenode heartbeat which makes the zookeeper fail to get the pingback in time and marks the NN as dead and elects a new leader and that keeps happening in a loop. So check those ZK logs to ensure time is set correctly and is in sync!
Thank you for the elaborative answers . In my cluster we have 2 ZKFC and 3 Zookeeper and 2 NN.
I have been through the NN and ZKFC logs and i found the below errors , can we conclude something from it ?
NN error logs -
ZKFC logs -
From the logs, I see connectivity loss and that's precisely what's causing the NN switch. Remember in my earlier posting the importance of Zk quorum!
Your NN and losing Connection to the ZK so the NN that loses active connection is causing the ZK to elect a new leader and that's happening in a loop
Caused by : java.net.SocketTimeoutException: 5000 millis timeout while waiting for channel to be ready for read
I would start by checking FW I see you are on Ubuntu so ensure the FW is disabled across the Cluster.
The root cause of a Socket Timeout is a connectivity failure between the machines, so try the usual process
Check your hostname resolution DNS or /etc/hosts should be in sync and another important thing is all your host time should be in sync.
can you share the value of Core-site.xml parameter ha.zookeeper.quorum
Thank you for the descriptive response .
Kindly find the value for the asked property as below -
Core-site.xml parameter ha.zookeeper.quorum -