zookeeper server + Unable to load database on disk
We have kafka cluster with 3 nodes , each kafka include zookeeper server and schema registry
We get the following error on one of the zookeeper server
[2019-11-12 07:44:20,719] ERROR Unable to load database on disk (org.apache.zookeeper.server.quorum.QuorumPeer)
java.io.IOException: Unreasonable length = 198238896
at org.apache.jute.BinaryInputArchive.checkLength(BinaryInputArchive.java:127)
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:92)
at org.apache.zookeeper.server.persistence.Util.readTxnBytes(Util.java:233)
at org.apache.zookeeper.server.persistence.FileTxnLog$FileTxnIterator.next(FileTxnLog.java:629)
at org.apache.zookeeper.server.persistence.FileTxnSnapLog.restore(FileTxnSnapLog.java:166)
at org.apache.zookeeper.server.ZKDatabase.loadDataBase(ZKDatabase.java:223)
at org.apache.zookeeper.server.quorum.QuorumPeer.loadDataBase(QuorumPeer.java:601)
at org.apache.zookeeper.server.quorum.QuorumPeer.start(QuorumPeer.java:591)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:164)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:111)
at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
seems that some snapshot files under folder /opt/confluent/zookeeper/data/version-2 are corrupted
under folder version-2 , we have the following example files
many files as log.3000667b5
many files as snapshot.200014247
one file - acceptedEpoch
one file – currentEpoch
so the question is – how to start the zookeeper server
from my understanding we have two options , but not sure about them
one option is to move version-2 folder to other place as version-2_backup
and create new folder - version-2 under /opt/confluent/zookeeper/data
then start the zookeeper server and hope that snapshot will copied from other good active zookeeper server ?
second option is maybe to move version-2 folder to other place as version-2_backup , create new folder as - version-2
and copy all content from version-2 from good machine to the bad zookeeper server to version-2 , but I not sure if this is right option?
