Zookeeper not starting on one node inconsistent epoch

anand_raghavan — Tue, 20 Aug 2019 12:57:55 GMT

Dear All,

I recently upgraded our cluster (3 node not secure) from HDP 2.3 to 2.5. After the upgrade was finalized and when i was restarting some of the services I saw that zookeeper went down on one of the nodes with the following errors in the log

2016-10-04 05:38:06,427 - INFO [main:QuorumPeer@1038] - initLimit set to 10 2016-10-04 05:38:06,439 - INFO [main:FileSnap@83] - Reading snapshot /hadoop/zookeeper/version-2/snapshot.4400000000 2016-10-04 05:38:06,556 - ERROR [main:QuorumPeer@530] - Unable to load database on disk java.io.IOException: The current epoch, 45 is less than the accepted epoch, 32 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:527) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:473) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) 2016-10-04 05:38:06,558 - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally java.lang.RuntimeException: Unable to run quorum server at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:531) at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:473) at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:153) at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111) at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) Caused by: java.io.IOException: The current epoch, 45 is less than the accepted epoch, 32 at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:527) ... 4 more 2016-10-04 05:50:32,198 - INFO [main:FourLetterWordMain@43] - connecting to localhost 2181 2016-10-04 06:01:35,510 - INFO [main:FourLetterWordMain@43] - connecting to localhost 2181

I saw that there is a JIRA https://issues.apache.org/jira/browse/ZOOKEEPER-2307, do i need to apply this patch. Has anyone else faced this problem.

Thanks,

Anand